What is MinIO?

MinIO is a high-performance, S3-compatible object storage solution designed for AI/ML workloads. With enterprise-grade features, MinIO provides a robust foundation for organizations seeking private cloud storage solutions

Prerequisites

Before connecting MinIO to Datature, ensure you have:

• An existing MinIO deployment with proper configuration

Please make sure to meet the MinIO prerequisites before you set up self-managed object storage. For prerequisites, refer to MinIO’s Object Storage for Linux or other preferred options in the MinIO documentation.
For respective options, please refer to MinIO’s ‘Deployment Checklists’ (Example) for guidance for production-readiness of MinIO deployments

• Network accessibility to MinIO endpoints

The setup depends on your current infrastructure setup. Please refer to your respective service provider for more information.

📘

Note:

Please contact us at support@datature if you need IP-whitelisting

• Valid SSL certificates for secure connections
• Properly configured DNS settings
• Public endpoints (if accessing from external networks)
• Read-only Permissions configured on MinIO
  • Please note that MinIO default readonly policy is insufficient for the connection and sync. You will need to add s3:ListBucketto allow us to list the objects in your bucket. Example below:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation",
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                 ## Restrict your respective bucket for least-priviledge best practice
                "arn:aws:s3:::your-bucket-name",
                "arn:aws:s3:::your-bucket-name/*"
            ]
        }
    ]
}

Security Best Practices for MinIO Bucket Access

Least privilege access

• Grant only the minimum required permissions needed for Datature operations:
  • s3:GetBucketLocation
  • s3:GetObject
  • s3:ListBucket
  • Restrict to specific bucket only
• Avoid using default admin credentials
• Generate separate Access and Secret Keys for the specific user account for Datature integration
  • You can set a custom policy for the access key that further restricts the permissions granted to users authenticating only to the desired bucket that you want to connect to Datature.
  • Please refer to MinIO documentation for more information.
• Rotate keys periodically (recommended every 90 days)
• Avoid sharing or reusing access keys across different integrations or services

Setup the Connection on Datature Nexus Platform

Starting in your chosen project page, follow the steps below to set up the connection and sync your data.

• Navigate to the Dataset tab in your project
• Select "Connect to External Buckets"

• Choose "MinIO" from the storage options

• Enter your connection details:

  • MinIO Connection Name - Your preferred Connection Name
  • MinIO Bucket Name - Your MinIO Bucket Name
  • MinIO Connection Endpoint - MinIO Endpoint URL
  • Under the Advanced Option, you can specify:
    • Folder Prefix if you would like to restrict the connection sync to a specific folder.
    • Path Style - Enable path-based URL format for file access by specifying bucket names within request paths rather than as endpoint subdomains.
  

• Next, input your credentials:

  • Access Key ID - MinIO Access Key or username
  • Access Secret Key - MinIO Secret Key or password

• Click Next and wait for the connection testing to complete

• Successful Connection will show:

• Click ‘Sync Now’ and head to the ‘Connection Manager’ tab after closing the Popup window by clicking ‘I Understand’. You will see the progress of the sync for your dataset.

• After the syncing is completed, the images from your bucket will appear on the dashboard.

How MinIO Integration Works and General Precautions

MinIO Bucket Connectivity Functionality

• Read-Only Access: Our platform only reads image and video metadata from your MinIO bucket. Your actual data remains in your MinIO storage, ensuring that the bucket acts as the master dataset. Any changes made to your dataset on our platform, including deleting assets, will not affect the original data in your MinIO bucket.
• Sync Behavior: If updates are made to your MinIO bucket (e.g., adding or removing assets) and you sync the connection on our platform, the latest changes will be reflected. However, changes made on our platform (e.g., annotations or labels) are stored separately and do not modify the source data in your bucket.
• Multiple Buckets: If you connect multiple MinIO buckets and sync them on the same Assets page, all images from the synced buckets will be added into the same project.

📘

Note:

If there are different objects with similar names in multiple buckets, the objects will be overwritten.

MinIO Bucket Connectivity Relating to Quota Usage

• Quota Impact: Every asset synced from your MinIO bucket counts toward your account's image or video quota. Ensure that your account has sufficient quota before syncing large datasets. To check or upgrade your quota, visit the "Resource Usage" section of your account.
• Suggestions: Before syncing, review your asset count and expected usage to avoid exceeding your quota limits.

Asset Requirements

• Synced images must now satisfy the following new criteria because we cannot strip EXIF tags from synced assets:
  • The asset must have no EXIF orientation tag, or the image must have an EXIF orientation of 1 (i.e., it is already in the upright position)
• Any MP4 files are supported, but with the following restrictions:
  • major_brand or compatible_brand must include at least one of the supported brands: isom, iso2, mp41, mp42
  • Pixel format (pix_fmt): yuv420p
  • Video stream metadata should not contain any mentions of 4:4:4 or 4:2:2
  • Number of frames (nbframes): Should be similar to r_frame_rate timebase duration_ts
  • [If Present] sample_aspect_ratio and SAR should either be 1:1 or any corresponding 1-to-1 ratio.

To check the above metadata in your videos, you can use ffprobe to retrieve the information:

# For Ubuntu-based systems
sudo apt-get install -y ffprobe
ffprobe <YOUR_VIDEO>.mp4

These restrictions ensure that the videos can play in all supported browsers in the annotator. Typical MP4 files should be able to meet these requirements.

Since we do not hold your video data on our platform, externally-synced video assets will not be modified in any way. This means that they will retain their original dimensions, quality, and audio, if any.