Writing Core Octostar Apps
Core Octostar apps are no different than other apps, as they use the same framework, conventions and libraries detailed in the rest of this Section. However, to ensure these apps are maintanable, portable and reliable, especially given that different versions of these apps could be deployed under different Octostar environments, there are a few recommendations to be followed.
App Repository Structure​
The basic structure for a core app should be as follows:
.
├── main.sh # entry point for the app
├── localdev.py # entry point for testing the app locally
├── .env # sensitive credentials for testing the app locally. MUST NOT BE TRACKED
├── manifest.yaml # octostar app manifest
├── config.yaml # exposed configs for the app (optional)
├── Dockerfile # app image definition
├── requirements.txt # python requirements file (optional)
├── build.sh # all requirements and downloads should be executed here
├── README.md # provide clear docs to users
├── .gitignore # ignore common python by-products and the .env file
├── .github
├── workflows
├── tag.yaml # create and push a new image for the app
The main.sh file is the entry point for the app, which should define the running process(es) (e.g. streamlit server, fastapi server, etc.).
App developers should test the app locally by running the localdev.py file (or equivalent in other languages), which should read environment variables from the .env file and set the OS_DEV_MODE flag to true before launching either main.sh or a specific app process/main file.
The manifest.yaml file should contain at least title, icon and description of the app. It must specify a unique alias among all apps and it must define the app itself as its image, e.g.
alias: my-app
image: octostar/app.my-app:version
This is because core Octostar apps should be versioned and built as Docker images (More on this in the Creation of a Docker Image section).
The build.sh file should refer to all external dependencies and install them, including python libraries, javascript packages and downloadable contents. It will be executed as part of the Dockerfile definition, ensuring the application comes packaged with all its dependencies and therefore will be reliable and can be installed without internet access. This file should also install the requirements.txt, if present. It is good practice for python dependencies to be frozen (pip freeze), and the same applies to apt-get calls (which should refer to OS-specific versions according to the base image defined in the Dockerfile).
PRE-DOWNLOADING PYTHON MODELS
Often, python libraries related to data science will download models or contents when they are imported or specific functions are called for the first time. In order to force this download so that it is part of the built image, it is perfectly okay for thebuild.sh file to run a python snippet, like so:
python -c "f
from spacy_download import load_spacy
load_spacy('en_core_web_sm') # will download the en_core_web_sm if not found
"
A tag.yaml file should be also provided for a GitHub action which will deploy a new Docker image for the app (same name as the one defined in the manifest.yaml) to the Octostar Docker Registry. This action should trigger whenever a commit is tagged (where a tag = a version of the app).
Finally, the README.md should provide a user-friendly guide to the app in terms of its features, basic and advanced usage, installation and developer guide (More on this in the Readme section). In particular, it should refer to the config.ini file and how its settings can change the behavior of the app.
APP SECRETS
It is crucial that sensitive secrets (such as API keys) are not committed to the repository, ever. The basic setup for developers should be to always rely on the .env file to store app secrets locally, or the Secrets Manager in production. Therefore the .env file should NEVER be tracked, and secrets should NEVER be written directly into source files.
Creation of a Docker Image​
The Dockerfile should have the following structure:
FROM {your_image}
WORKDIR /app
RUN apt-get update && apt-get install -y python3-venv
RUN python3 -m venv /app/venv
COPY . /app
RUN chmod +x build.sh
RUN source /app/venv/bin/activate && ./build.sh
ENTRYPOINT ["/bin/bash", "-c", "source /app/venv/bin/activate && /app/main.sh"]
EXPOSE $CODE_SERVER_PORT
Which will copy all app files tracked by git, execute the build.sh in a virtual environment and set main.sh as the app entrypoint. Note this image should be launched only via a GitHub action and NOT locally, as it could include files that are not tracked by git (like the .env file).
Typically (but this is not mandatory) the FROM image is one of:
octostar/streamlit-apps-gpu(python 3.10 w/ CUDA support)octostar/streamlit-apps-no-gpu(python 3.11)
BE CAREFUL OF THE PYTHON VERSION
The python version of the base image is binding, particularly in the case of GPU-enabled Docker images. The application should be tested locally on the same python version as the image version to ensure compatibility of all libraries.
Versioning and GitHub Integration​
GitHub integration allows to automate the creation of a Docker image every time a new version of the app is ready. To do so, we need to:
- tag to the remote a new version of the app, like so:
cd /path/to/the/app
git tag 1.2.3-dev4 # or any other version
git push origin --tags
The action will trigger upon pushing a new tag to the remote, building and uploading a new docker image for the app, which is then referenced in the manifest.yaml file.
Typically, the two main branches for a core app should be dev (default branch) and main, where the former releases stable versions (e.g. v0.1.2) and the latter development/experimental versions (e.g. v0.1.1-dev.1).
The Apps Workspace​
Once an app has been pushed as a docker image to the Octostar registry, in order to install and launch the app we only need to provide three files:
.
├── manifest.yaml # octostar app manifest
├── config.yaml # exposed configs for the app (optional)
├── README.md # provide clear docs to users
In particular, it is good practice that the app version in the manifest is frozen to a specific one, e.g.:
image: octostar/app.my-app:1.2.3
Core apps are packaged together (as a workspace with folders each with the above files) into the ws.apps repository. Upon creating a new app (or updating an existing core app), the workspace itself should be updated with one or more commits and tagged with a new version, exactly as one does for an app. This will create a new docker image for the workspace itself, such that newly deployed Octostar environments will fetch the new version of this workspace and, therefore, of the core apps.
Readme​
The README.md is a crucial documentation tool for Octostar applications, as it allows fellow developers and analysts to understand how your application works, what features it provides and how it is meant to be deployed and used. It is recommended that all readmes follow the same general structure (which of course can be expanded upon):