Alex. CLI

Alex. is a decentralized archival platform that preserves human history and culture digitally. It leverages the immutable and permanent data storage of Arweave. Institutions and creators are incentivized to publicly archive digital content by allowing anyone to become sponsors of their digital artifacts in what we call pools.

This documentation allows anyone to start their own pool and earn $AR. For a more high level overview of the Alex Archive, please visit our about page.

Main concepts

  • Pool: a topic, event, or idea that can be digitally and publicly archived.
  • Artifact: Permanently preserved digital content that can be documents, books, images, audio, or videos. Artifacts are deployed as atomic assets to Arweave and distributed to the contributors based on the amount of $AR contributed to a pool.
  • Pool operator: A user that creates and runs a pool that mines artifacts for a topic/event/idea of their choice. The pool operator earns $AR by declaring a percentage of the contributions to be kept.
  • Pool contributor: A user that contributes $AR to a pool in order to store artifacts on Arweave and pay the pool operator for doing the work of mining artifacts. Pool contributors receive artifacts sent to their wallet for their contributions.

Requirements

arcpool requires NodeJS (v18+) NPM and Git installed.

An Arweave web wallet with a small amount of $AR on it is needed to deploy your pool. You can get one here:

Installing Alex. CLI

To install arcpool run npm install --global arcpool in your terminal.

npm install --global arcpool

Usage: arcpool [commands] [options]

Options                                                     Description
--clear                                                     Clear local search index for pool
--content-moderation                                        Use content moderation on twitter mining
--control-wallet <wallet file>                              Specifies a wallet to use in the pool creation
--dname <string>                                            Specifies the daemon name to stop
--image <path to image file>                                Specifies an image to use for pool
--mention-tag <username>                                    Username for twitter or reddit with --method user
--method <user / mention / subreddit / search>              Subcategory within source such as user
--search-term <search term>                                 Search term to mine
--source <twitter / wikipedia / reddit>                     Specifies the data source
--subreddit <subreddit>                                     Subreddit to mine


Commands                                                    Description
balance <pool id>                                           Check the Bundlr and Arweave balance for the pool wallet
create <pool id>                                            Create a pool using pools.json
dlist                                                       list all daemon mining processes
dstop <daemon name>                                         Stop a daemon mining process by name
fund <pool id>                                              Fun the bundlr wallet for a pool
help                                                        Display help text
init <pool id>                                              Initialize pools.json
mine <pool id>                                              Mine artifacts for a given pool

Configuring a pool

Create a directory for the pool configurations and wallets. Name the directory anything you want.

To create a pool you must do the following steps:

  1. Initiate the pool
  2. Configure the pool
  3. Connect wallet client and create pool

Initiate the pool

In the terminal you will create a directory for pool configurations and wallets.

  1. Create directory
mkdir {DIRECTORY_NAME}
  1. Change into the directory that you created
cd {DIRECTORY_NAME}
  1. Create a pools.json file.
arcpool init {POOL_ID}
  • {POOL_ID} refers to a string of your choosing which will identify the pool locally for the client.
  • This string is only used in CLI commands to identify which pool you are working with and this will not be visible on Alex.
  • You will edit this pools.json file and it will also be modified by the client so do not delete it after running the create and mine commands, unless you need to start over.

Example:

mkdir alex-test
cd alex-test
arcpool init test-alex

Edit the pool.json configuration

Below is the pools.json file that has been initiated that now needs to be configured.

{
    "POOL_ID": {
        "appType": "Alex-Archiving-Pool-v1.4",
        "contracts": {
            "nft": {
                "id": "",
                "src": ""
            },
            "pool": {
                "id": "",
                "src": ""
            }
        },
        "state": {
            "owner": {
                "pubkey": "",
                "info": ""
            },
            "controller": {
                "pubkey": "",
                "contribPercent": 0
            },
            "title": "Pool Title eg. Russia Ukraine War",
            "description": "Paragraph/HTML for long pool description",
            "briefDescription": "Paragraph/HTML for short pool description",
            "link": "",
            "rewards": "",
            "image": "",
            "timestamp": ""
        },
        "walletPath": "",
        "bundlrNode": "https://node2.bundlr.network",
        "keywords": [
            "keyword1",
            "keyword2"
        ],
        "twitterApiKeys": {
            "consumer_key": "",
            "consumer_secret": "",
            "token": "",
            "token_secret": "",
            "bearer_token": ""
        },
        "clarifaiApiKey": "",
        "topics": [
            "history",
            "technology"
        ],
        "redditApiKeys":{
            "username": "",
            "password": "",
            "appId": "",
            "appSecret": ""
        }
    }
}

Configure the pools.json file to generate your pool.

  • state.title is the title of your pool on the home page of Alex.
  • state.description is a long description of your pool on the pool details page of Alex., under the 'About' header. It can contain text and/or HTML.
  • state.briefDescription is a brief description of your pool on the home page of Alex.
  • keywords is a list of the main keywords to track in the mining process. This is the core driving data that instructs the mining programs of what to pull from mining sources such as Twitter, Wikipedia, Reddit, or Nostr
  • topics A list of more general topics the pool fits into these generate ANS110 Topic tags in the data. Examples: history, funny, humor, science.
  • controller.contribPercent is a reward percentage of the contributions. This value can be from 0 - 100 and represents the total percentage of contributions that go to your control wallet for mining. If the value of controller.contribPercent is 10, then 10% of all contributions will go to your control wallet as a reward. This percentage will appear in the pool user interface to let contributors know how much of their contribution will go to the operator.
  • controller.pubkey is the public key for the control wallet that the percentage of contributions declared in controller.contribPercent will be sent to.

Configure Twitter API Keys

  • Get Twitter API credentials here. (Get elevated access for better mining, but not mandatory)
  • Enter credentials into twitterApiKeys
    • consumer_key = API key in the Twitter developer platform
    • consumer_secret = Secret in the Twitter developer platform
    • token = Access Token in the Twitter developer platform
    • token_secret = Access Token Secret in the Twitter developer platform
    • bearer_token = Bearer Token in the Twitter developer platform
    • clarifaiApiKey if you plan to use content moderation on tweets in the mining process, you can get an api key from Clarifai and put it here. This will filter out explicit content from being mined into the pool. This is a bit expensive and unnecessary in most situations but if you are finding a lot of explicit content in the pool it may be of value to you.

Configure Reddit API Keys

  • redditApiKeys login or create a Reddit account and then get API access here
  • Use username and password from Reddit
  • Insert appId and appSecret received from Reddit

Creating a pool

To create a pool you only need to run 1 command with a few arguments passed in, including the name from when you initialized the pool, path to wallet you created, and a path to your pool header image.

arcpool create <POOL_NAME> --control-wallet <PATH_TO_WALLET.json> --image <PATH_TO_IMAGE>

For example:

arcpool create init-test-alex --control-wallet ../wallet.json --image ../pool-image.jpg

Take note of the line at the top of the logs for your pool wallets seed phrase. Inside your working directory should be a wallets directory. This is where your pool collection wallet is stored and should be kept safe, along with the seed phrase logged by the CLI. Write down your seed phrase.

Do not give this to anyone. Without this you have no way to recover your wallet should anything happen.

*** Write the following seed phrase down ***

 this will be your pool wallet seed phrase do not give out
 
*** THERE IS NO WAY TO RECOVER YOUR SEED PHRASE SO WRITE IT DOWN AND KEEP IT OUT OF OTHERS HANDS ***

If your pool has been successfully created, you can now navigate to https://alex.arweave.dev/#/pools and view your new pool.

To add another pool, follow the same steps as above in the same directory with the pools.json

Funding a pool wallet

At the end of the creating a pool process the cli should ask you if you want to contribute funds from your control wallet. This is recommended as it will allow you to begin mining immediately. When the prompt asks, enter a decimal amount of AR less than what you have in your wallet and hit enter. The funding portion will take some time because it is waiting for the transactions to process on the blockchain. Wait for the program to finish.

If you decide to not fund the pool from your control wallet, you will need contributions to start mining. This can be a small amount to get started just to trigger the CLI to start transferring your funds to your pools Bundlr instance. We will only need to do this once and as more contributions are made it will be updated automatically as we run the mining service.

Find and navigate to your pool in https://alex.arweave.dev/#/pools and click the contribute button. Contribute a small amount of $AR to your collection and wait for that contribution to display in the UI. Once that has registered the funds to the pool, go back to your terminal and run:

arcpool fund POOL_NAME

For example:

Please note that funding a Bundlr instance can take up to 30 minutes. To check the status of the funding you can use the command:

arcpool balance POOL_NAME

Once you see that you have Bundlr funds you can proceed to the mining process.

Mining artifacts

Before Mining: double check the keywords in the pools.json to prevent unwanted artifacts in your pool.

The mining process can begin in the directory containing the pools.json by running these commands.

Twitter

Mine tweets (runs 100 tweets at a time)

arcpool mine POOL_NAME --source twitter

Mine all tweets based on user and tag

For example: mine all tweets with "@thealexarchive #TOPIC"

arcpool mine POOL_NAME --source twitter --method mention --mention-tag "@thealexarchive #TOPIC"

Mine all tweets ever from a particular user

For example: mine all tweets from SBF_FTX

  • Do not include the @ in the --username value
arcpool mine POOL_NAME --source twitter --method user --username SBF_FTX

Wikipedia

Mine a single Wikipedia article related to the given keywords in config

arcpool mine POOL_NAME --source wikipedia

Reddit

Mine Reddit posts by search term

arcpool mine wildlife --source reddit --method search --search-term america

Mine Reddit posts by subreddit

arcpool mine wildlife --source reddit --method subreddit --subreddit webdev

Mine Reddit posts by username

arcpool mine wildlife --source reddit --method user --username exampleusername

Nostr

Mine common Nostr threads for posts related to the keywords

arcpool mine <POOL_ID> --source nostr

Checking mining process

After a few minutes of mining you can navigate back to your pool on Alex and see that the artifact count is growing and contributors are having artifacts minted to their addresses as expected.

Daemon mode mining

If you wish to continually run a mining process, use daemon mode by passing the --d flag to any of the above mining commands. Daemon mode is built on top of pm2.

Mine tweets into the pool from above, still runs for 20 seconds but the daemon mode will continue restarting the program infinetly. Note the --d flag.

arcpool mine POOL_NAME --source twitter --d

To view all the daemon mode mining processes:

arcpool dlist

Output will look similar to:

daemon processes -
pid: 0    pm_id: 0    name: POOL_NAME    status: running

Stop a pools daemon process by name:

arcpool dstop --dname POOL_NAME

To view logs for the mining processes install pm2:

npm install --global pm2

Stream the logs:

pm2 logs