[ Hybrid(s) of machine learning and irc3 framework ] + netspanning [ irc/matrix ]
Go to file
.[d]. 73f6113340 update 2023-08-04 01:24:54 -05:00
.vscode update5 2023-03-26 18:01:42 -05:00
screenshots v2.666-6 2022-09-05 15:30:29 -05:00
storage update6 2023-03-27 07:26:58 -05:00
README.md Update 'README.md' 2023-03-29 16:19:41 +00:00
maple update 2023-08-04 01:24:54 -05:00

[ MAPLE ]


video

MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMh+MMMMMMMMMMMMMMhsMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMm/ oMMMMMMMMMMMMMMm +NMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMy`  yMMMMMMMMMMMMMMM- -mMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMs+dMMMMMMMMMM+    sMMMMMMMMMMMMMMM-  `dMMMMMMMMMMms/NMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMM+ .omMMMMMM:     -MMMMMMMMMMMMMMo    `yMMMMMMMy: `dMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMM-   /dMMM+       sMMMMMMMMMMMMh      `hMMMNo`   sMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMd     :dm        `mMMMMMMMMMMN.       .NNo`    .MMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMM:      -         :MMMMMMMMMMs         :`      sMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMs                 ymNMMMMMNm.                 NMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMy                    `-/-`                   .MMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMo                                            .NMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMNh+.                                               :sdMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMhso+:.                                                      `-/+syMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMM-                                                                 dMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMM`   `.:+/.                                              `/s+:.    sMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMNo   -oms.             .//-`               `:/:`            `+md+`  .hMMMMMMMMMMMMMMM
MMMMMMMMMMMMMNs`                        .odNdo.       .ohmd+`                        :dMMMMMMMMMMMMM
MMMMMMMMMMMNo`                             ..           .-                             :hMMMMMMMMMMM
MMMMMMMMMd+`                                                                             -sNMMMMMMMM
MMMMMMNs-                         `..                          `/-.                        `+dMMMMMM
MMMNy:                       ./sdNMMMh:                      `sNMMMNds/.                      .odMMM
MM+                       :ymMMMMMMMMMMh.                   +NMMMMMMMMMMmo-                      /NM
MMMh:                  .sNMMMMMMMMMMMMMMN-                `hMMMMMMMMMMMMMMMm+`                 :hMMM
MMMMMd:`       ``-:+shmMMMMMMMMMMMMMMMMMMN.               hMMMMMMMMMMMMMMMMMMMmhs+/-..``````./dMMMMM
MMMMMMMMMNNNNNNMMMMMMMMMMMMMMMMMMMMMMMMMMMo              .MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMy              .MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMN.             /MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMN+`         `+NMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMNs.    -hMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMdyymMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM

[ SUMMARY ]

this project is a hybrid of hybrids - ( m4pl1mp - https://git.tcp.direct/decoded/m4pl1mp )
-
this version spans across different networks and not limited to irc, it connects matrix,
telegram, discord and offers up a newer way of unifying those services in one core process.
to summarize it's g1mp from security @ blackhats.ru, maple from tcpdirect @ ircd.chat and
a parallel process across many networks with those same services in hydradized modular ai.


[ SCREENSHOTS ]


screenshots


[ API KEYS ] - get these or some before you start

  • some of maple's services require api developer keys for certain sites, it is recommended you get them all
  • at minimal you should have a youtube, cryptocompare, possibly twitter key. all of these api keys are free
  • to setup and the only one that would really ever cost anything is openai using gpt3 backend and that isn't
  • even required, maple gpt2 runs offline and doesn't require openai services for language models like davici.

[ Relevant urls pertaining to the developer api keys and server credentials ]



[ PREREQUISITES ]

- [ substitute apt for the package manager of your choice ]

- [ virtualenv could be substituted for pyenv/pipenv or no virtualenv et al ]

- [ environmental credentials in env/bin/activate could alternatively be hardcoded ]

  • 1 apt install python3 - *note: python3.9 is ideal
  • 2 apt install python3-pip
  • 3 python3 -m pip install virtualenv

[ MAPLE ]

- Setup

  • git clone https://git.tcp.direct/decoded/maple.git && cd maple && bash maple

- Usage

  • ./maple - this will ideally setup/start/stop maple services

[ NOTES ]

[ Append your credentials for developer/service api keys into ~/maple/env/bin/activate ]

cat ~/maple/env/bin/activate
..
export CRYPTOCOMPARE_KEY=your_cryptocompare_developer_key                        # <- CRYPTOCOMPARE_KEY
export NICKSERV_USERNAME=your_nickserv_username                                  # <- IRCD NICKSERV USER
export NICKSERV_PASSWORD=your_nickserv_password                                  # <- IRCD NICKSERV PASS
export BOT_SASL_USERNAME=$NICKSERV_USERNAME                                      # <- IRCD NICKSERV USER
export BOT_SASL_PASSWORD=$NICKSERV_PASSWORD                                      # <- IRCD NICKSERV PASS
export MATRIX_HOMESERVER=https://matrix.org                                      # <- MATRIX HOMESERVER
export MATRIX___USERNAME=your_matrix_username                                    # <- MATRIX USERNAME
export MATRIX___PASSWORD=your_matrix_password                                    # <- MATRIX PASSWORD
export DEVELOPER_KEY=1394823190182390182382383215382158321                       # <- YOUTUBE API KEY
export CONSUMER_KEY=2151235132512351235123512351325231                           # <- TWITTER API KEY
export CONSUMER_SECRET=514512521345234523452345234523452345234523452             # <- TWITTER API KEY
export ACCESS_TOKEN_KEY=24513429875209348502934850294898348034850293485203948592 # <- TWITTER API KEY
export ACCESS_TOKEN_SECRET=523490582034985203948520394804884820934850923485      # <- TWITTER API KEY
export OPENAPI_KEY=AeJWbyak16wjHZ8L4TAbGIlVk31YwwAyz-GsezWCJ7Wkh7BIFTO           # <- OPENAI API KEY
export DISCORD__MAPLE=295326962352954038                                         # <- DISCORD API KEY
export DISCORD__SCHAN=295326962352954038                                         # <- DISCORD API KEY
export DISCORD__TOKEN=AT2Sh4g.G1M9vNTOXmIaQzdvn5b6QzQMTyTBX6COJONUIRjLWi5UX0j294 # <- DISCORD API KEY
export TELEGRAM_TOKEN=AHh4DA9FgE3vs1SfAnx8975t85J30mU925GhJWcoUBBo7              # <- TELEGRAM API KEY
export HYDRA_DESIGNATION=dupe                                                    # <- DEBUG CORE OR DUPE
export HYDRA_HOME=#b0tsh0p                                                       # <- DEBUG HOME CHANNEL


THE API KEYS / THEIR VARIABLES / THEIR USAGE AND FUNCTIONALITY IN MAPLE ENVIRONMENT

  • 1 env/bin/activate - the actual credentials to be exported into the local environment
  • 2 relevant plugin - reads these particular environmental variable exports
  • 3 the source code - where the credentials are read back to as runtime only variables

[ this method can be hardened with a cipher, but at some point a key is required ]


   1         export CRYPTOCOMPARE_KEY=your_cryptocompare_developer_key
   2 plugins.crypto_plugin
   3         CRYPTOCOMPARE_KEY=os.environ['CRYPTOCOMPARE_KEY']

   1         export NICKSERV_USERNAME=your_nickserv_username
   1         export NICKSERV_PASSWORD=your_nickserv_password
   2 plugins.auth_custom_plugin
   3         NICKSERV_USERNAME=os.environ['NICKSERV_USERNAME']
   3         NICKSERV_PASSWORD=os.environ['NICKSERV_PASSWORD']

   1         export BOT_SASL_USERNAME=also_your_nickserv_username
   1         export BOT_SASL_PASSWORD=also_your_nickserv_password
   2 plugins.sasl_custom_plugin
   3         BOT_SASL_USERNAME=os.environ['BOT_SASL_USERNAME']
   3         BOT_SASL_PASSWORD=os.environ['BOT_SASL_PASSWORD']

   1        export MATRIX_HOMESERVER=https://matrix.org
   1        export MATRIX___USERNAME=your_matrix_username
   1        export MATRIX___PASSWORD=your_matrix_password
   2 plugins.net_matrix_plugin
   3        MATRIX_HOMESERVER=os.environ['MATRIX_HOMESERVER']
   3        MATRIX___USERNAME=os.environ['MATRIX___USERNAME']
   3        MATRIX___PASSWORD=os.environ['MATRIX___PASSWORD']

   1        export OPENAPI_KEY=AeJWbyak16wjHZ8L4TAbGIlVk31YwwAyz-GsezWCJ7Wkh7BIFTO
   2 plugins.openai_plugin
   3        OPENAPI_KEY = os.environ['OPENAPI_KEY']

   1        export DISCORD__MAPLE=295326962352954038
   1        export DISCORD__SCHAN=295326962352954038
   1        export DISCORD__TOKEN=AT2Sh4g.G1M9vNTOXmIaQzdvn5b6QzQMTyTBX6COJONUIRjLWi5UX0j294
   2 plugins.net_discord_plugin
   3        DISCORD__SCHAN=int(os.environ['DISCORD__SCHAN'])
   3        DISCORD__MAPLE=int(os.environ['DISCORD__MAPLE'])
   3        DISCORD__TOKEN=str(os.environ['DISCORD__TOKEN'])

   1        export TELEGRAM_TOKEN=AHh4DA9FgE3vs1SfAnx8975t85J30mU925GhJWcoUBBo7
   2 plugins.net_telegram_plugin
   3        TELEGRAM_TOKEN=os.environ['TELEGRAM_TOKEN']

   1        export DEVELOPER_KEY=1394823190182390182382383215382158321
   2 plugins.youtube_plugin
   3        DEVELOPER_KEY = os.environ['DEVELOPER_KEY']

   1        export CONSUMER_KEY=2151235132512351235123512351325231
   1        export CONSUMER_SECRET=514512521345234523452345234523452345234523452
   1        export ACCESS_TOKEN_KEY=24513429875209348502934850294898348034850293485203948592
   1        export ACCESS_TOKEN_SECRET=523490582034985203948520394804884820934850923485
   2 plugins.twitter_plugin
   3        CONSUMER_KEY = os.environ['CONSUMER_KEY']
   3        CONSUMER_SECRET = os.environ['CONSUMER_SECRET']
   3        ACCESS_TOKEN_KEY = os.environ['ACCESS_TOKEN_KEY']
   3        ACCESS_TOKEN_SECRET = os.environ['ACCESS_TOKEN_SECRET']

[ TODO ]

  • the initial maple setup build will be affected by user selective services - e.g. maybe no discord or telegram, and maybe not matrix either, not every user will require 100% of the services in full fledge operation.
  • crypto_plugin also has it's own developer key, cryptocompare needs to be added to the rest of the devkeys
  • tool_colors_plugin, colorform - this plugin has a formatting glitch and is still offline in a few modules due to chaos variable formatting randomness - it requires better sanitization and exception handling alternatives.
    • it should be in every plugin replacing the print function when fully sanitized by formatting schemes and center piece to the main admin heads up display pane.
  • cleanup tools_bus_plugin, there was input/output bridge messaging - this code will be re merged
  • cleanup tools_dim_plugin, there is lingering code that is either offline or antiquated but interwoven in #s
  • openai plugin is written and functions, the setup automation has omitted it's incorporation until baselined
  • convert escaped irc color codes back to terminal codes for ansi_plugin.py

[ CHANGELOG/VERSION HISTORY ]

v2.9

  • overhaul - too much to name

v2.8

  • boombox - youtube bug listing spotify.
  • boombox - open.spotify.com url was not pulling page title correctlyl
  • dsa - updated heavily
  • txt2img - an intermediate for accesssing local stable diffusion webui api
  • seen - was not acrewing activity days
  • hydra - will readout connected hydras
  • dims - updated to reflect permissions
  • cmd_irc - updated to reflect permissions
  • base - added a bunch of mass commands to mass deop / halfop / voice / etc.
  • urlgrabber - updated a contentlength abort on prefetch bug

v2.7

  • user tailorization
  • automated setup

v2.666-3

  • changed the directory structure for ease of use with the hydra system during development
  • ive got the hydra system split down at minimal 3.. 1 that stays up no matter what, another that runs
  • all of services developed up to this point, and the last one is kind of like the core, it's stripped
  • of services because there is already a bot running services but this one can develop new services and
  • stop,start,restart,crash etc.. as much as the developer would like with out sacraficing being online
  • or known services going offline..

v2.666-2

  • plugins/ansi_plugin.py is an ansi recapture utility.. basically if someone is pumping ansi art into a
  • channel this will copy it 1:1 but for the intent of using 'ansipants', converting the graphic to an image
  • and then re-uploading the image rather than the ansi/utf8 to discord/matrix/telegram due to the fact they
  • easily or properly display ansi graphics but they show images.

v2.666-1

  • plugins/net_hydra_plugin.py is a multiband wrapper that i use as a headless hydra code/debug logic.
  • meaning a core, that stripped of services will always be online, so no matter what a presensce is online.
  • anything non core is a dupe, aka hydra.. one may run the ai one machine, and another may just be that im
  • coding something and dont want service to go offline and yet i myself require them to code. so that is
  • why and how this hydra logic is used, and it because of features of sasl authentication that it's done.

v2.666

  • fixed bridge responses double echoing
  • simplified the calling of some plugins
  • incorporated a multihead wrapper to keep services running

v2.5

  • telegram/discord/matrix/irc now netspanned ( always more to do )
  • the routerio logic mostly finished in plugins/tool_bus_plugin.py
  • the way inter-process threads are handled are now speakable/observable
  • maple ai is now online, it's very fast now, almost immediate response

v2.4

  • telegram netspanned
  • experimenting with color coding

v2.3

  • discord netspanned

v2.2

  • matrix netspanned

v2.1

  • tried pyenv+pipenv, ended up up reverting back to pure virtualenv for consistency.

[ BACKSTORY ]

[ CURRENT NETSPANNED PROTOCOLS ]

[ THE BASE HYBRID ]

[ BOT BASE CODES ]


[ MATRIX PROTOCOL - OLM/MEGOLM E2E DOUBLE RATCHET CRYPTOGRAPHY NOTES ]

OLM LIBRARY - FOR - MATRIX PROTOCOL REGARDING MAPLE'S NETSPAN PROTOCOL
- MAPLE NETSPAN PLUGIN -> https://git.tcp.direct/decoded/maple/src/branch/main/storage/bot/plugins/net_matrix_plugin.py
- NIO PYTHON MODULE ( MATRIX API CLIENT ) -> https://github.com/poljar/matrix-nio 
- OLM LIBRARY -> https://gitlab.matrix.org/matrix-org/olm

[WTF-U-SAY-ME]

MAPLE COMMUNICATES VIA A NETSPANNED 'MATRIX PROTOCOL', HER PLUGIN REQUIRES A MATRIX API CLIENT WHICH IN THIS CASE IS A PYTHON MODULE NAMED 'NIO' TO FACILITATE THAT. WHERE MATRIX CHANNEL/USER ENCRYPTION IS USED, AN ADDITIONAL (BASE) LIBRARY IS REQUIRED TO SIMPLIFY THIS 'OLM/MEGOLM DOUBLE RATCHET END-2-END ENCRYPTION'. FOR 'OLM' -A SYSTEM LIBRARY AND A LANGUAGE BINDINGS TO THAT SYSTEM LIBRARY ARE BOTH REQUIRED. THIS TEST SYSTEM IS DEBIAN, THE SYSTEM LIBRARY WE BUILD IS 'LIBOLM.SO', OUR SCRIPTED PROGRAMMING LANGUAGE IS PYTHON V3.9 AND THIS REQUIRES A PORTING OF SORTS OR COMPATIBILITY LAYER KNOWN AS BINDINGS TO ACCESS THAT SYSTEM LIBRARY. POINT BEING- WITHOUT 'OLM' OUR SERVICE AI BOT MAPLE IS OTHERWISE BLIND AND CAN NOT SEE TO ACKNOWLEDGE 'ANY' OUTBOUND COMMUNICATION FROM 'ANY' POINTS OF ORIGIN WHERE USERS/CHANNELS HAVE THAT E2E ENCRYPTION ENABLED.

OLM LIBRARY BUILD NOTES (TESTED ON DEBIAN)

[PREREQUISITE]

  • sudo apt install cmake

[OBJECTIVE]

assuming you are using maple bot, we are going to skip over that bot's setup directly to where we are dropping in the 'olm' library for e2e functionality

[PROCESS]

A - get the maple bot src, enter it's root directory

B - within that root directory get the 'olm' src, also enter it's root directory

C - build 'olm' system library

  • cmake . -Bbuild && cmake --build build
		dr1p in 🌐 SAISABER in olm on  master via △ v3.18.4 via 🌙 via 🐍 v3.9.2 via 🐦
		 cmake . -Bbuild && cmake --build build
		-- The CXX compiler identification is GNU 10.2.1
		-- The C compiler identification is GNU 10.2.1
		-- Detecting CXX compiler ABI info
		-- Detecting CXX compiler ABI info - done
		-- Check for working CXX compiler: /usr/bin/c++ - skipped
		-- Detecting CXX compile features
		-- Detecting CXX compile features - done
		-- Detecting C compiler ABI info
		-- Detecting C compiler ABI info - done
		-- Check for working C compiler: /usr/bin/cc - skipped
		-- Detecting C compile features
		-- Detecting C compile features - done
		-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY
		-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY - Success
		-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY
		-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY - Success
		-- Performing Test COMPILER_HAS_DEPRECATED_ATTR
		-- Performing Test COMPILER_HAS_DEPRECATED_ATTR - Success
		-- Configuring done
		-- Generating done
		-- Build files have been written to: /home/dr1p/maple/olm/build
		Scanning dependencies of target olm
		[  1%] Building CXX object CMakeFiles/olm.dir/src/account.cpp.o
		[  3%] Building CXX object CMakeFiles/olm.dir/src/base64.cpp.o
		[  5%] Building CXX object CMakeFiles/olm.dir/src/cipher.cpp.o
		[  7%] Building CXX object CMakeFiles/olm.dir/src/crypto.cpp.o
		[  9%] Building CXX object CMakeFiles/olm.dir/src/memory.cpp.o
		[ 11%] Building CXX object CMakeFiles/olm.dir/src/message.cpp.o
		[ 13%] Building CXX object CMakeFiles/olm.dir/src/pickle.cpp.o
		[ 15%] Building CXX object CMakeFiles/olm.dir/src/ratchet.cpp.o
		[ 16%] Building CXX object CMakeFiles/olm.dir/src/session.cpp.o
		[ 18%] Building CXX object CMakeFiles/olm.dir/src/utility.cpp.o
		[ 20%] Building CXX object CMakeFiles/olm.dir/src/pk.cpp.o
		[ 22%] Building C object CMakeFiles/olm.dir/src/sas.c.o
		[ 24%] Building C object CMakeFiles/olm.dir/src/ed25519.c.o
		[ 26%] Building C object CMakeFiles/olm.dir/src/error.c.o
		[ 28%] Building C object CMakeFiles/olm.dir/src/inbound_group_session.c.o
		[ 30%] Building C object CMakeFiles/olm.dir/src/megolm.c.o
		[ 32%] Building CXX object CMakeFiles/olm.dir/src/olm.cpp.o
		[ 33%] Building C object CMakeFiles/olm.dir/src/outbound_group_session.c.o
the		[ 35%] Building C object CMakeFiles/olm.dir/src/pickle_encoding.c.o
olm		[ 37%] Building C object CMakeFiles/olm.dir/lib/crypto-algorithms/aes.c.o
library	[ 39%] Building C object CMakeFiles/olm.dir/lib/crypto-algorithms/sha256.c.o
is		[ 41%] Building C object CMakeFiles/olm.dir/lib/curve25519-donna/curve25519-donna.c.o
------> [ 43%] Linking CXX shared library libolm.so
named  	[ 43%] Built target olm
libolm	Scanning dependencies of target test_ratchet
.		[ 45%] Building CXX object tests/CMakeFiles/test_ratchet.dir/test_ratchet.cpp.o
so		[ 47%] Linking CXX executable test_ratchet
		[ 47%] Built target test_ratchet
		Scanning dependencies of target test_base64
		[ 49%] Building CXX object tests/CMakeFiles/test_base64.dir/test_base64.cpp.o
		[ 50%] Linking CXX executable test_base64
		[ 50%] Built target test_base64
		Scanning dependencies of target test_sas
		[ 52%] Building CXX object tests/CMakeFiles/test_sas.dir/test_sas.cpp.o
		[ 54%] Linking CXX executable test_sas
		[ 54%] Built target test_sas
		Scanning dependencies of target test_crypto
		[ 56%] Building CXX object tests/CMakeFiles/test_crypto.dir/test_crypto.cpp.o
		[ 58%] Linking CXX executable test_crypto
		[ 58%] Built target test_crypto
		Scanning dependencies of target test_olm_decrypt
		[ 60%] Building CXX object tests/CMakeFiles/test_olm_decrypt.dir/test_olm_decrypt.cpp.o
		[ 62%] Linking CXX executable test_olm_decrypt
		[ 62%] Built target test_olm_decrypt
		Scanning dependencies of target test_group_session
		[ 64%] Building CXX object tests/CMakeFiles/test_group_session.dir/test_group_session.cpp.o
		[ 66%] Linking CXX executable test_group_session
		[ 66%] Built target test_group_session
		Scanning dependencies of target test_message
		[ 67%] Building CXX object tests/CMakeFiles/test_message.dir/test_message.cpp.o
		[ 69%] Linking CXX executable test_message
		[ 69%] Built target test_message
		Scanning dependencies of target test_megolm
		[ 71%] Building CXX object tests/CMakeFiles/test_megolm.dir/test_megolm.cpp.o
		[ 73%] Linking CXX executable test_megolm
		[ 73%] Built target test_megolm
		Scanning dependencies of target test_olm_using_malloc
		[ 75%] Building CXX object tests/CMakeFiles/test_olm_using_malloc.dir/test_olm_using_malloc.cpp.o
		[ 77%] Linking CXX executable test_olm_using_malloc
		[ 77%] Built target test_olm_using_malloc
		Scanning dependencies of target test_olm_sha256
		[ 79%] Building CXX object tests/CMakeFiles/test_olm_sha256.dir/test_olm_sha256.cpp.o
		[ 81%] Linking CXX executable test_olm_sha256
		[ 81%] Built target test_olm_sha256
		Scanning dependencies of target test_olm_signature
		[ 83%] Building CXX object tests/CMakeFiles/test_olm_signature.dir/test_olm_signature.cpp.o
		[ 84%] Linking CXX executable test_olm_signature
		[ 84%] Built target test_olm_signature
		Scanning dependencies of target test_olm
		[ 86%] Building CXX object tests/CMakeFiles/test_olm.dir/test_olm.cpp.o
		[ 88%] Linking CXX executable test_olm
		[ 88%] Built target test_olm
		Scanning dependencies of target test_session
		[ 90%] Building CXX object tests/CMakeFiles/test_session.dir/test_session.cpp.o
		[ 92%] Linking CXX executable test_session
		[ 92%] Built target test_session
		Scanning dependencies of target test_list
		[ 94%] Building CXX object tests/CMakeFiles/test_list.dir/test_list.cpp.o
		[ 96%] Linking CXX executable test_list
		[ 96%] Built target test_list
		Scanning dependencies of target test_pk
		[ 98%] Building CXX object tests/CMakeFiles/test_pk.dir/test_pk.cpp.o
		[100%] Linking CXX executable test_pk
		[100%] Built target test_pk

D - enter python root directory and build the 'olm' bindings for python to interact with the 'olm' system library

  • cd python && make
		dr1p in 🌐 SAISABER in olm/python on  master via 🐍 v3.9.2
		 make
		mkdir -p include/olm
		cc -E -DOLM_STATIC_DEFINE -I dummy -I ../include -o include/olm/olm.h ../include/olm/olm.h
		echo 'void *memset(void *s, int c, size_t n);' >> include/olm/olm.h
		cc -E -DOLM_STATIC_DEFINE -I dummy  -I ../include -o include/olm/pk.h ../include/olm/pk.h
		cc -E -DOLM_STATIC_DEFINE -I dummy -I ../include -o include/olm/sas.h ../include/olm/sas.h
		cc -E -DOLM_STATIC_DEFINE -I dummy -I ../include -o include/olm/error.h ../include/olm/error.h
		DEVELOP= python3 setup.py build
		make[1]: Entering directory '/home/dr1p/maple/olm/python'
		make[1]: Nothing to be done for 'headers'.
		make[1]: Leaving directory '/home/dr1p/maple/olm/python'
		running build
		running build_py
		creating build
		creating build/lib.linux-x86_64-3.9
		creating build/lib.linux-x86_64-3.9/olm
		copying olm/session.py -> build/lib.linux-x86_64-3.9/olm
		copying olm/account.py -> build/lib.linux-x86_64-3.9/olm
		copying olm/utility.py -> build/lib.linux-x86_64-3.9/olm
		copying olm/_finalize.py -> build/lib.linux-x86_64-3.9/olm
		copying olm/group_session.py -> build/lib.linux-x86_64-3.9/olm
		copying olm/_compat.py -> build/lib.linux-x86_64-3.9/olm
		copying olm/pk.py -> build/lib.linux-x86_64-3.9/olm
		copying olm/sas.py -> build/lib.linux-x86_64-3.9/olm
		copying olm/__version__.py -> build/lib.linux-x86_64-3.9/olm
		copying olm/__init__.py -> build/lib.linux-x86_64-3.9/olm
		running build_ext
		generating cffi module 'build/temp.linux-x86_64-3.9/_libolm.c'
		creating build/temp.linux-x86_64-3.9
		building '_libolm' extension
		creating build/temp.linux-x86_64-3.9/build
		creating build/temp.linux-x86_64-3.9/build/temp.linux-x86_64-3.9
		x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -ffile-prefix-map=/build/python3.9-RNBry
		6/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -ffile-prefix-map=/build/python3.9-RNBry6/python3.9-3.9.2=. -fstack-protector-strong
		-Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/
		python3.9 -c build/temp.linux-x86_64-3.9/_libolm.c -o build/temp.linux-x86_64-3.9/build/temp.linux-x86_64-3.9/_libolm.o -I../include
		x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -Wl,-z,relro -g -fwrapv -O2 -g -ffile-prefix-map=/build/python3.9-RNBry6/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.9/build/temp.linux-x86_64-3.9/_libolm.o -lolm -o build/lib.linux-x86_64-3.9/_libolm.abi3.so -L../build

note - the 'olm' system library that was built is here - ~/maple/olm/build/libolm.so.3.2.12

		dr1p in 🌐 SAISABER in maple on  main [?] via 🐍 v3.9.2
		 ls -al ~/maple/olm/build|grep libolm
		lrwxrwxrwx   11 dr1p  5 Oct 18:01 libolm.so -> libolm.so.3
		lrwxrwxrwx   16 dr1p  5 Oct 18:01 libolm.so.3 -> libolm.so.3.2.12
--->	.rwxr-xr-x 206k dr1p  5 Oct 18:01 libolm.so.3.2.12

note - the 'python' bindings are here - ~/maple/olm/python/build/lib.linux-x86_64-3.9

		dr1p in 🌐 SAISABER in python/build/lib.linux-x86_64-3.9 on  master
		 fd
		olm
		olm/__init__.py
		olm/__version__.py
		olm/_compat.py
		olm/_finalize.py
		olm/account.py
		olm/group_session.py
		olm/pk.py
		olm/sas.py
		olm/session.py
		olm/utility.py

E - enter maple's root directory to activate that virtual environment, this is so when we go to install these 'olm python bindings' they will be attached correctly. finally, we have to enable all these things we done for the 'nio' matrix api client

activate the maple virtualenv

  • cd ~/maple && source env/bin/activate
		dr1p in 🌐 SAISABER in ~ via 🐍 v3.9.2
		 cd ~/maple && source env/bin/activate

install our 'olm' bindings

  • cd ~/maple/olm/python && pip install .
		dr1p in 🌐 SAISABER in maple on  main [?] via 🐍 v3.9.2 (env) <- virtualenv activated 
		 cd ~/maple/olm/python && pip install .
		Processing /home/dr1p/maple/olm/python
		  Preparing metadata (setup.py) ... done
		Collecting cffi>=1.0.0
		  Using cached cffi-1.15.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (441 kB)
		Requirement already satisfied: future in /home/dr1p/maple/env/lib/python3.9/site-packages (from python-olm==3.2.12) (0.18.2)
		Collecting pycparser
		  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
		Building wheels for collected packages: python-olm
		  Building wheel for python-olm (setup.py) ... done
		  Created wheel for python-olm: filename=python_olm-3.2.12-cp39-cp39-linux_x86_64.whl size=133136 sha256=87e55620faaff38a54474fe7a230b2baa10d1371d2c339ee89bd26769c678478
		  Stored in directory: /tmp/pip-ephem-wheel-cache-694ae4yo/wheels/a4/f7/aa/838212f08c0e2322272e3054a1ed5a6d4f3d27e998a34f973d
		Successfully built python-olm
		Installing collected packages: pycparser, cffi, python-olm
		Successfully installed cffi-1.15.1 pycparser-2.21 python-olm-3.2.12

3 - to enable 'olm' is really just switch over from build of 'nio' to a different build of 'nio'. this new one is purposely based in this 'olm' library

  • pip install "matrix-nio[e2e]"
		dr1p in 🌐 SAISABER in olm/python on  master via 🐍 v3.9.2 (env) took 5s
		 pip install "matrix-nio[e2e]"
		Requirement already satisfied: matrix-nio[e2e] in /home/dr1p/maple/env/lib/python3.9/site-packages (0.20.0)
		Requirement already satisfied: pycryptodome<4.0.0,>=3.10.1 in /home/dr1p/maple/env/lib/python3.9/site-packages (from matrix-nio[e2e]) (3.15.0)
		Requirement already satisfied: aiohttp<4.0.0,>=3.7.4 in /home/dr1p/maple/env/lib/python3.9/site-packages (from matrix-nio[e2e]) (3.8.3)
		Requirement already satisfied: future<0.19.0,>=0.18.2 in /home/dr1p/maple/env/lib/python3.9/site-packages (from matrix-nio[e2e]) (0.18.2)
		Requirement already satisfied: aiohttp-socks<0.8.0,>=0.7.0 in /home/dr1p/maple/env/lib/python3.9/site-packages (from matrix-nio[e2e]) (0.7.1)
		Requirement already satisfied: h2<5.0.0,>=4.0.0 in /home/dr1p/maple/env/lib/python3.9/site-packages (from matrix-nio[e2e]) (4.1.0)
		Requirement already satisfied: unpaddedbase64<3.0.0,>=2.1.0 in /home/dr1p/maple/env/lib/python3.9/site-packages (from matrix-nio[e2e]) (2.1.0)
		Requirement already satisfied: jsonschema<5.0.0,>=4.4.0 in /home/dr1p/maple/env/lib/python3.9/site-packages (from matrix-nio[e2e]) (4.16.0)
		Requirement already satisfied: h11<0.13.0,>=0.12.0 in /home/dr1p/maple/env/lib/python3.9/site-packages (from matrix-nio[e2e]) (0.12.0)
		Requirement already satisfied: logbook<2.0.0,>=1.5.3 in /home/dr1p/maple/env/lib/python3.9/site-packages (from matrix-nio[e2e]) (1.5.3)
		Requirement already satisfied: aiofiles<0.7.0,>=0.6.0 in /home/dr1p/maple/env/lib/python3.9/site-packages (from matrix-nio[e2e]) (0.6.0)
		Collecting atomicwrites<2.0.0,>=1.4.0
		  Downloading atomicwrites-1.4.1.tar.gz (14 kB)
		  Preparing metadata (setup.py) ... done
		Requirement already satisfied: python-olm<4.0.0,>=3.1.3 in /home/dr1p/maple/env/lib/python3.9/site-packages (from matrix-nio[e2e]) (3.2.12)
		Requirement already satisfied: cachetools<5.0.0,>=4.2.1 in /home/dr1p/maple/env/lib/python3.9/site-packages (from matrix-nio[e2e]) (4.2.2)
		Collecting peewee<4.0.0,>=3.14.4
		  Downloading peewee-3.15.3.tar.gz (858 kB)
		     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 858.6/858.6 kB 10.3 MB/s eta 0:00:00
		  Preparing metadata (setup.py) ... done
		Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /home/dr1p/maple/env/lib/python3.9/site-packages (from aiohttp<4.0.0,>=3.7.4->matrix-nio[e2e]) (2.1.1)
		Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /home/dr1p/maple/env/lib/python3.9/site-packages (from aiohttp<4.0.0,>=3.7.4->matrix-nio[e2e]) (4.0.2)
		Requirement already satisfied: frozenlist>=1.1.1 in /home/dr1p/maple/env/lib/python3.9/site-packages (from aiohttp<4.0.0,>=3.7.4->matrix-nio[e2e]) (1.3.1)
		Requirement already satisfied: attrs>=17.3.0 in /home/dr1p/maple/env/lib/python3.9/site-packages (from aiohttp<4.0.0,>=3.7.4->matrix-nio[e2e]) (22.1.0)
		Requirement already satisfied: multidict<7.0,>=4.5 in /home/dr1p/maple/env/lib/python3.9/site-packages (from aiohttp<4.0.0,>=3.7.4->matrix-nio[e2e]) (6.0.2)
		Requirement already satisfied: yarl<2.0,>=1.0 in /home/dr1p/maple/env/lib/python3.9/site-packages (from aiohttp<4.0.0,>=3.7.4->matrix-nio[e2e]) (1.8.1)
		Requirement already satisfied: aiosignal>=1.1.2 in /home/dr1p/maple/env/lib/python3.9/site-packages (from aiohttp<4.0.0,>=3.7.4->matrix-nio[e2e]) (1.2.0)
		Requirement already satisfied: python-socks[asyncio]<3.0.0,>=2.0.0 in /home/dr1p/maple/env/lib/python3.9/site-packages (from aiohttp-socks<0.8.0,>=0.7.0->matrix-nio[e2e]) (2.0.3)
		Requirement already satisfied: hpack<5,>=4.0 in /home/dr1p/maple/env/lib/python3.9/site-packages (from h2<5.0.0,>=4.0.0->matrix-nio[e2e]) (4.0.0)
		Requirement already satisfied: hyperframe<7,>=6.0 in /home/dr1p/maple/env/lib/python3.9/site-packages (from h2<5.0.0,>=4.0.0->matrix-nio[e2e]) (6.0.1)
		Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /home/dr1p/maple/env/lib/python3.9/site-packages (from jsonschema<5.0.0,>=4.4.0->matrix-nio[e2e]) (0.18.1)
		Requirement already satisfied: cffi>=1.0.0 in /home/dr1p/maple/env/lib/python3.9/site-packages (from python-olm<4.0.0,>=3.1.3->matrix-nio[e2e]) (1.15.1)
		Requirement already satisfied: pycparser in /home/dr1p/maple/env/lib/python3.9/site-packages (from cffi>=1.0.0->python-olm<4.0.0,>=3.1.3->matrix-nio[e2e]) (2.21)
		Requirement already satisfied: idna>=2.0 in /home/dr1p/maple/env/lib/python3.9/site-packages (from yarl<2.0,>=1.0->aiohttp<4.0.0,>=3.7.4->matrix-nio[e2e]) (2.10)
		Building wheels for collected packages: atomicwrites, peewee
		  Building wheel for atomicwrites (setup.py) ... done
		  Created wheel for atomicwrites: filename=atomicwrites-1.4.1-py2.py3-none-any.whl size=6943 sha256=ac56bd16b57e62ad7e6ba608848839761d0ccc2159f0e49666509f442f56ffda
		  Stored in directory: /home/dr1p/.cache/pip/wheels/2a/b5/06/d2f34584f352e4af7d1d7ac1baf38b5d24142c8044cd024fd5
		  Building wheel for peewee (setup.py) ... done
		  Created wheel for peewee: filename=peewee-3.15.3-py3-none-any.whl size=134170 sha256=9ad2e9a6171911168d33d15c66960d15103fc3553675a67ca680269745c0805b
		  Stored in directory: /home/dr1p/.cache/pip/wheels/18/97/9a/f80ba10d0b6e96c9b77abfc8b8de18c71694b624011f571d9a
		Successfully built atomicwrites peewee
		Installing collected packages: peewee, atomicwrites
		Successfully installed atomicwrites-1.4.1 peewee-3.15.3

F - note: i built but did not relocate to install the olm system library before starting maple and she crashed(obviously), but we can see the python bindings work and it is requiring a system library at all so now at least we know everything is working

		dr1p in 🌐 SAISABER in ~ via 🐍 v3.9.2 (env)
		 deactivate && cd ~/maple && bash maple
		located jq
		located nc

		[ PREREQUISITES CHECK - PASSED ]
		[ running hydra_dupe__services ]
		[ hydra_dupe__services bot added to background as a job. pid: 11739 ]

		dr1p in 🌐 SAISABER in maple on  main [?] via 🐍 v3.9.2
		 Traceback (most recent call last):
		  File "/home/dr1p/maple/bot_maple.py", line 11, in <module>
		    from nio import AsyncClient, MatrixRoom, RoomMessageText
		  File "/home/dr1p/maple/env/lib/python3.9/site-packages/nio/__init__.py", line 9, in <module>
		    from .client import *
		  File "/home/dr1p/maple/env/lib/python3.9/site-packages/nio/client/__init__.py", line 3, in <module>
		    from .base_client import Client, ClientConfig
		  File "/home/dr1p/maple/env/lib/python3.9/site-packages/nio/client/base_client.py", line 36, in <module>
		    from ..crypto import ENCRYPTION_ENABLED, DeviceStore, OlmDevice, OutgoingKeyRequest
		  File "/home/dr1p/maple/env/lib/python3.9/site-packages/nio/crypto/__init__.py", line 26, in <module>
		    from .sessions import (  # isort:skip
		  File "/home/dr1p/maple/env/lib/python3.9/site-packages/nio/crypto/sessions.py", line 23, in <module>
		    import olm
		  File "/home/dr1p/maple/env/lib/python3.9/site-packages/olm/__init__.py", line 24, in <module>
		    from .utility import ed25519_verify, OlmVerifyError, OlmHashError, sha256
		  File "/home/dr1p/maple/env/lib/python3.9/site-packages/olm/utility.py", line 39, in <module>
		    from _libolm import ffi, lib  # type: ignore
		ImportError: libolm.so.3: cannot open shared object file: No such file or directory

G - copy 'olm' library to /usr/lib and create the symbolic links to it

 sudo cp ~/maple/olm/build/libolm.so.3.2.12 /usr/lib
 sudo ln -s /usr/lib/libolm.so.3.2.12 /usr/lib/libolm.so.3
 sudo ln -s /usr/lib/libolm.so.3.2.12 /usr/lib/libolm.so
	dr1p in 🌐 SAISABER in maple on  main [?] via 🐍 v3.9.2
	 ls -al /usr/lib/libolm*
	lrwxrwxrwx   25 root  5 Oct 19:14 /usr/lib/libolm.so -> /usr/lib/libolm.so.3.2.12
	lrwxrwxrwx   25 root  5 Oct 19:14 /usr/lib/libolm.so.3 -> /usr/lib/libolm.so.3.2.12
	.rwxr-xr-x 206k root  5 Oct 19:13 /usr/lib/libolm.so.3.2.12

H - test - start maple and see if the olm'd version of 'nio' crashes, it doesn't so we are done

dr1p in 🌐 SAISABER in maple on  main [?] via 🐍 v3.9.2
 ./maple
located jq
located nc

[ PREREQUISITES CHECK - PASSED ]
[ running hydra_dupe__services ]
[ hydra_dupe__services bot added to background as a job. pid: 12081 ]

dr1p in 🌐 SAISABER in maple on  main [?] via 🐍 v3.9.2
 <<< system_status__info >>> [ main loop ] - startup
<<< ________botio_class >>> [ instantiated ]
...

FUTURE RELEVANCE

python-olm
==========

Python bindings for Olm.

The specification of the Olm cryptographic ratchet which is used for peer to
peer sessions of this library can be found [here][4].

The specification of the Megolm cryptographic ratchet which is used for group
sessions of this library can be found [here][5].

An example of the implementation of the Olm and Megolm cryptographic protocol
can be found in the Matrix protocol for which the implementation guide can be
found [here][6].

The full API reference can be found [here][7].

# Accounts

Accounts create and hold the central identity of the Olm protocol, they consist of a fingerprint and identity
key pair. They also produce one time keys that are used to start peer to peer
encrypted communication channels.

## Account Creation

A new account is created with the Account class, it creates a new Olm key pair.
The public parts of the key pair are available using the identity_keys property
of the class.

```python
>>> alice = Account()
>>> alice.identity_keys
{'curve25519': '2PytGagXercwHjzQETLcMa3JOsaU2qkPIESaqoi59zE',
 'ed25519': 'HHpOuFYdHwoa54GxSttz9YmaTmbuVU3js92UTUjYJgM'}

One Time keys

One time keys need to be generated before people can start an encrypted peer to peer channel to an account.

>>> alice.generate_one_time_keys(1)
>>> alice.one_time_keys
{'curve25519': {'AAAAAQ': 'KiHoW6CIy905UC4V1Frmwr3VW8bTWkBL4uWtWFFllxM'}}

After the one time keys are published they should be marked as such so they aren't reused.

>>> alice.mark_keys_as_published()
>>> alice.one_time_keys
{'curve25519': {}}

Pickling

Accounts should be stored for later reuse, storing an account is done with the pickle method while the restoring step is done with the from_pickle class method.

>>> pickle = alice.pickle()
>>> restored = Account.from_pickle(pickle)

Sessions

Sessions are used to create an encrypted peer to peer communication channel between two accounts.

Session Creation

>>> alice = Account()
>>> bob = Account()
>>> bob.generate_one_time_keys(1)
>>> id_key = bob.identity_keys["curve25519"]
>>> one_time = list(bob.one_time_keys["curve25519"].values())[0]
>>> alice_session = OutboundSession(alice, id_key, one_time)

Encryption

After an outbound session is created an encrypted message can be exchanged:

>>> message = alice_session.encrypt("It's a secret to everybody")
>>> message.ciphertext
'AwogkL7RoakT9gnjcZMra+y39WXKRmnxBPEaEp6OSueIA0cSIJxGpBoP8YZ+CGweXQ10LujbXMgK88
xG/JZMQJ5ulK9ZGiC8TYrezNYr3qyIBLlecXr/9wnegvJaSFDmWDVOcf4XfyI/AwogqIZfAklRXGC5b
ZJcZxVxQGgJ8Dz4OQII8k0Dp8msUXwQACIQvagY1dO55Qvnk5PZ2GF+wdKnvj6Zxl2g'
>>> message.message_type
0

After the message is transfered, bob can create an InboundSession to decrypt the message.

>>> bob_session = InboundSession(bob, message)
>>> bob_session.decrypt(message)
"It's a secret to everybody"

Pickling

Sessions like accounts can be stored for later use the API is the same as for accounts.

>>> pickle = session.pickle()
>>> restored = Session.from_pickle(pickle)

Group Sessions

Group Sessions are used to create a one-to-many encrypted communication channel. The group session key needs to be shared with all participants that should be able to decrypt the group messages. Another thing to notice is that, since the group session key is ratcheted every time a message is encrypted, the session key should be shared before any messages are encrypted.

Group Session Creation

Group sessions aren't bound to an account like peer-to-peer sessions so their creation is straightforward.

>>> alice_group = OutboundGroupSession()
>>> bob_inbound_group = InboundGroupSession(alice_group.session_key)

Group Encryption

Group encryption is pretty simple. The important part is to share the session key with all participants over a secure channel (e.g. peer-to-peer Olm sessions).

>>> message = alice_group.encrypt("It's a secret to everybody")
>>> bob_inbound_group.decrypt(message)
("It's a secret to everybody", 0)

Pickling

Pickling works the same way as for peer-to-peer Olm sessions.

>>> pickle = session.pickle()
>>> restored = InboundGroupSession.from_pickle(pickle)