# Software Environments

To help GitLab CI/CD build jobs faster and keep our infrastructure secure, we incorporate several software components in CI/CD environments.

## Local Docker Mirror
A local [Docker registry mirror](https://docs.docker.com/docker-hub/mirror/) offers faster image retrieval and reduced bandwidth usage by caching images locally. 
It improves reliability, ensuring access to images even if external connections are disrupted, and enhances control and security within private networks.

### Benefits of a Local Docker Registry Mirror
- Speed: Faster access to Docker images through local caching.
- Bandwidth Efficiency: Less network data transfer, conserving bandwidth.
- Reliability: Constant image availability, even with network issues.
- Security and Control: Improved security with internal network image transfers.

### Usage
After setting up a registry server, client systems can connect to and use this Docker registry server.
When configured, any image pull requests from the client system will go through this registry server.
This setup streamlines the distribution and sharing of images, reducing network traffic and improving download speeds.

#### Client System Setup (cpu-runner)
The Docker registry server on `etri-storage` is configured for efficient image management. 
To connect your client system to this setup, simply follow these steps, keeping in mind that adjustments can be made to meet specific needs.

To integrate with the `etri-storage` Docker registry, the Docker daemon on the client system requires configuration. 
This is done by editing the `/etc/docker/daemon.json` file.

Add the following configuration:
```json
{
    "registry-mirrors": ["http://storage.ones-ai.lab:6000"],
    "exec-opts": ["native.cgroupdriver=systemd"],
    "log-driver": "json-file",
    "log-opts": {
        "max-size": "100m"
    },
    "storage-driver": "overlay2"
}
```

Save the file and restart the Docker daemon to apply these changes: `sudo systemctl restart docker`

After this, your Docker commands will be directed to the local registry mirror automatically. Simply execute any docker command and it should work: `docker run hello-world`

### Setting up Docker Registry Mirror

You can configure `docker-compose` to setup a Docker registry mirror. In our case, we employ `etri-storage` server for this.

File Path(etri-storage) : `/docker-mirror/docker-compose.yml`

```yaml
version: "3"
services:
  dockerregistrymirror:
    container_name: etri-docker-registry
    image: registry:latest
    ports:
      - "6000:5000"
    environment:
      - REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io
      - REGISTRY_PROXY_USERNAME=###
      - REGISTRY_PROXY_PASSWORD=###
    restart: always
```

Note: Add `REGISTRY_PROXY_USERNAME` and `REGISTRY_PROXY_PASSWORD` if pulling private images is required.

### Testing Registry

#### Test Registry Connectivity:
To confirm that the mirror is operational:
```bash
curl http://storage.ones-ai.lab:6000/v2/_catalog
```

#### Pull an Image (e.g., nginx):
For instance, to pull the nginx image:
```bash
docker pull nginx
```

#### Recheck the registry:
Verify that the image is now in your local mirror:
```bash
curl http://storage.ones-ai.lab:6000/v2/_catalog
```

#### Logs and monitoring
Regularly check logs for any operational insights or issues:
```bash
docker logs [DOCKER REGISTRY NAME] # Replace with your Docker registry container name, e.g., etri-docker-registry
```

## MINIO: Local Object Storage
[MinIO](https://github.com/minio/minio) is an open-source, high-performance local object storage system. 
It's compatible with `Amazon S3 , emphasizing self-hosted, private cloud environments for efficient management of large-scale unstructured data like photos, videos, and logs. 
This solution stands out for its scalability and security in personalized storage setups.

### Usage

#### Connect
```bash
mc alias set etri-storage [IP Address] [ACCESS_KEY] [SECRET_KEY]
```
#### Check Server Info:
```bash
mc admin info etri-storage
```

#### Create User
```bash
mc admin user add etri-storage [ACCESS_KEY] [SECRET_KEY]
```

#### Attach Policy:
```bash 
mc admin policy attach etri-storage readwrite --user=[ACCESS_KEY]
```

#### Set Alias for New User:

```bash
mc alias set ones-ai [IP Address] [ACCESS_KEY] [SECRET_KEY]
```

#### Create Bucket

```bash
mc mb ones-ai/nestc-pub
```
#### Upload Files
```bash
mc cp test.txt ones-ai/nestc-pub
```

#### Download Files

```bash
mc cp ones-ai/nestc-pub/test.txt ./local-directory
```

#### List Files in Bucket
```bash
mc ls ones-ai/nestc-pub
```

#### Remove Files in Bucket
```bash
mc rm ones-ai/nestc-pub/test.txt
```

### Using in Pipelines

In your pipeline, you can use MinIO storage using `mc`.

```yaml
publish:
  stage: publish
  image:
    name: minio/mc
    entrypoint: [""]
  before_script:
    - mc alias set $MINIO_ALIAS $MINIO_SERVER_URL $ACCESS_KEY $SECRET_KEY
  script:
    - mc rm $MINIO_ALIAS/$S3_BUCKET/ --recursive
    - mc cp build_ci $MINIO_ALIAS/$S3_BUCKET/ --recursive
  when: manual
  allow_failure: true
```
The usage of MinIO's CLI is very similar to AWS S3. [MinIO CLI Documentation](https://min.io/docs/minio/linux/reference/minio-mc.html)

- `$MINIO_ALIAS`: ones-ai
- `$MINIO_SERVER_URL`: cpu-runner (ip address)

### MinIO Docker Configuration
To configure MinIO with Docker, use the provided `docker-compose.yml` file.

File Path(etri-storage): `/minio/docker-compose.yml`

```yaml
version: '3'
services:
  minio:
    image: minio/minio:latest
    command: server /data --console-address ":8080"
    container_name: etri-storage
    environment:
      MINIO_ROOT_USER: xxxx
      MINIO_ROOT_PASSWORD: xxxx
    restart: always

    healthcheck:
      test: ["CMD", "mc", "ready", "local"]
      interval: 5s
      timeout: 5s
      retries: 5

    ports:
      - "80:9000"
      - "8080:9001"
    volumes:
      - minio-data:/data

volumes:
  minio-data:
```

## Alpacon

Alpacon provides secure access to our servers, allowing you to effortlessly log in to the system and access the servers for which you have authorization.

### Using Web

Open a Web browser, and visit https://ones-ai.alpacon.io. Upon logging in, you'll encounter a list of our servers. Choose a server from the list and establish a connection by clicking the `Websh` button.

### Using CLI

Install Alpacon CLI using the [instructions](https://github.com/alpacanetworks/alpacon-cli) for your platform.

#### Login

```bash
alpacon login
```

This command will prompt you to input username and password. For the server URL, use https://ones-ai.alpacon.io.

#### Connect

List servers that you have access:

```bash
alpacon servers ls
```

Connect to a server you want to access:

```bash
alpacon websh compute
```

```
Please wait until [compute] becomes connected...
Websh for [compute] became ready.

eunyoung@compute:~$ 
```

For root access:

```bash
alpacon websh -r compute
```