adding zed-lake project #1

Merged
calvin.house merged 1 commits from development into main 2025-10-13 20:38:43 +00:00
5 changed files with 721 additions and 1 deletions

View File

@@ -1,2 +1,5 @@
# Projects
# Table of Contents
- Zed/Zui projects
- [zed-lake](https://cave.phoe-nix.xyz/calvin.house/Projects/src/branch/main/zed-lake)
- [Zed Lake provisioning using CloudFormation]()

383
zed-lake/README.md Normal file
View File

@@ -0,0 +1,383 @@
# Create a Remote Zed Lake
> #### Checklist of resources
> - An SSH key pair
> - A cloud-config template
> - An AWS Account and permissions to create resources
> - A VPC
> - A Security Group
> - An AMI (Amazon Linux should fit in the free tier too)
> - An EC2 instance in the free tier (think t2.micro or t3.small)
> - If this is for production, right size your EC2 instance
## Prep Work
Create the SSH key pair. (ED25519 is used due to shortened key length and stronger security but RSA is still viable)
```
ssh-keygen -t ed25519 -f ~/.ssh/zed -C "zed lake"
```
Add a password if you desire, then add your key to the *ssh-agent*.
```
ssh-add .ssh/zed
```
And enter your password if you opted for a password.
You should have some output like:
```
Identity added: /home/your-username/.ssh/zed (zed lake)
```
## Create the ```cloud-config``` template
We'll go over the template in pieces, then look at the complete file.
This template is based on RHEL-based Linux distributions, but can easily be tailored to a different distribution
#### The ```users``` section
The user section is where you can add all user accounts and the relevant configuration of those users:
```
#cloud-config
users:
- name: zed
lock_passwd: false
passwd: some-passwd-hashed
ssh_authorized_keys:
- your-ssh-PUBLIC-KEY-here
shell: /bin/bash
groups: wheel
```
Here we have:
- The ```#cloud-config``` top line let's ```cloud-init```. That line is necessary for ```cloud-init``` to process the file.
- ```name``` is the username
- ```lock_passwd: false``` makes sure the user we're creating isn't locked. For this to remain true, the user needs a...
- ```passwd``` a password for said user. The ```openssl passwd -6``` command is great for allowing you to create a salted password
- ```ssh_authorized_keys:``` add the ***public key*** of your zed user here
- ```shell``` is the default login shell for the *zed* user
- ```groups``` is the groups the zed user belongs to
Optionally, the line ```sudo: ['ALL=(ALL) NOPASSWD: ALL']``` can be added.
This will allow the *zed* user to issue commands as root without prompting for a password. Care should be taken if this option is added to the config.
#### The ```write files``` section
We can leverage ```cloud-init``` to write a couple files:
```
write_files:
- path: /etc/ssh/sshd_config
content: |
Protocol 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
SyslogFacility AUTH
LogLevel INFO
LoginGraceTime 120
PasswordAuthentication no
PermitRootLogin no
StrictModes yes
PubkeyAuthentication yes
IgnoreRhosts yes
HostbasedAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no
PrintLastLog yes
TCPKeepAlive yes
AcceptEnv LANG LC_*
UsePAM yes
- path: /usr/lib/systemd/system/zed-lake.service
content: |
[Unit]
Description=Zed Lake Service
After=network.target
[Service]
ExecStart=/opt/Zui/resources/app.asar.unpacked/zdeps/zed serve -l :9867 -lake /home/zed/.config/Zui/lake -log.level=info -log.filemode=rotate -log.path=/home/zed/.config/Zui/logs/zlake.log
Restart=always
User=zed
[Install]
WantedBy=multi-user.target
```
This section tells ```cloud-init``` to do the following:
- We're writing a new ```sshd_config``` to define some rules, mostly to enforce public key authentication only and no root login
- We're creating a [systemd service](https://documentation.suse.com/smart/systems-management/html/systemd-setting-up-service/index.html) for our ```zed serve``` command to run automatically. At a high level we're:
- Telling the zed serve binary to listen on port 9867 on all interfaces:
- We'd usually define an IP here but we're not sure what IP AWS will assign us
- This can be changed once we know the IP
- Defining the lake
- Defining the log level
- Defining log rotation
- Defining the log path
#### The ```packages``` section
We can have packages installed ```cloud-init``` update the OS repositories and install packages during provisioning:
```
package_update: true
package_upgrade: true
packages:
- plocate
- unzip
- tar
- vim
- firewalld
```
Here we're updating package repositories and any packages that are on the system already, chiefly ```firewalld```.
#### The run ```runcmd``` section
This section will all define commands we want to run during the ```cloud-init``` process. This will round out the template and complete the setup.
```
runcmd:
- systemctl enable --now firewalld
- firewall-cmd --permanent --add-port 9867/tcp
- firewall-cmd --reload
- dnf install -y https://github.com/brimdata/zui/releases/download/v1.18.0/Zui-1.18.0.x86_64.rpm
- mkdir -p /home/zed/.config/Zui/lake /home/zed/.config/Zui/plugins/brimcap/storage/root /home/zed/.config/Zui/logs
- chown zed:zed -R /home/zed/.config
- find /opt/Zui/resources/app.asar.unpacked/zdeps/suricata -exec chmod go+w {} \;
- /opt/Zui/resources/app.asar.unpacked/zdeps/suricata/suricataupdater
- systemctl daemon-reload
- systemctl enable --now zed-lake.service
```
Here ```cloud-init``` is:
- Enabling the firewalld service (it usually has its preset as enabled but this will ensure that)
- Adding TCP port ***9867***
- Reloading firewalld
- Installing the Zui app:
- The Zui app has all the binaries we need for a zed lake, however the [zed](https://www.brimdata.io/download/) binary can be installed alone
- Some further [configuration](https://github.com/brimdata/brimcap/wiki/Custom-Brimcap-Config) for packet analysis
- Creating some directories for Zed. Some of these you saw in the ```systemd``` service we created earlier in the template
- Setting ownership back to our zed user. The actions taken by ```cloud-init``` are performed by root, so we'll need to restore ownership or our service will fail to start
- Configuration to enable packet analysis
- Reloading all daemons on the system, as we created a new one for the ```zed-lake.service```
- Enabling the ```zed-lake.service``` to auto-start
The template in its entirety should look like:
```
#cloud-config
users:
- name: zed
lock_passwd: false
passwd: some-passwd-hashed
ssh_authorized_keys:
- your-ssh-PUBLIC-KEY-here
shell: /bin/bash
groups: wheel
write_files:
- path: /etc/ssh/sshd_config
content: |
Protocol 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
SyslogFacility AUTH
LogLevel INFO
LoginGraceTime 120
PasswordAuthentication no
PermitRootLogin no
StrictModes yes
PubkeyAuthentication yes
IgnoreRhosts yes
HostbasedAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no
PrintLastLog yes
TCPKeepAlive yes
AcceptEnv LANG LC_*
UsePAM yes
- path: /usr/lib/systemd/system/zed-lake.service
content: |
[Unit]
Description=Zed Lake Service
After=network.target
[Service]
ExecStart=/opt/Zui/resources/app.asar.unpacked/zdeps/zed serve -l :9867 -lake /home/zed/.config/Zui/lake -log.level=info -log.filemode=rotate -log.path=/home/zed/.config/Zui/logs/zlake.log
Restart=always
User=zed
[Install]
WantedBy=multi-user.target
package_update: true
package_upgrade: true
packages:
- plocate
- unzip
- tar
- vim
- firewalld
runcmd:
- systemctl enable --now firewalld
- firewall-cmd --permanent --add-port 9867/tcp
- firewall-cmd --reload
- dnf install -y https://github.com/brimdata/zui/releases/download/v1.18.0/Zui-1.18.0.x86_64.rpm
- mkdir -p /home/zed/.config/Zui/lake /home/zed/.config/Zui/plugins/brimcap/storage/root /home/zed/.config/Zui/logs
- chown zed:zed -R /home/zed/.config
- find /opt/Zui/resources/app.asar.unpacked/zdeps/suricata -exec chmod go+w {} \;
- /opt/Zui/resources/app.asar.unpacked/zdeps/suricata/suricataupdater
- systemctl daemon-reload
- systemctl enable --now zed-lake.service
```
With the template created, we can move to AWS.
## The Cloud Side
> #### AWS Tasks
>- Create a VPC
>- Create a security group and security group rules
>- Create a subnet
>- Create an Internet Gateway (IGW)
>- Create a Route table and route to allow internet access for the subnet
>- Create the EC2 instance
#### Creating the ```VPC```
Probably the easiest part of our setup.
```
aws ec2 create-vpc --cidr-block 10.10.0.0/16 --tag-specifications ResourceType="vpc",Tags=['{Key=Name,Value=zed_vpc}']
```
Record the ```VpcId``` from the output, we'll need it again shortly.
Now we need to create a subnet within our VPC
```
aws ec2 create-subnet \
--vpc-id your-vpc-id \
--cidr-block your-cidr-block \
--tag-specifications ResourceType=subnet,Tags=[{'Key=Name,Value=zed_subnet}']
```
Record the ```SubnetId```. That'll be used shortly as well.
***Note: While tagging isn't necessary it helps to have descriptions of resources. So tag responsibly***.
***Another Note: If you're using a bash shell, make sure to place quotes around your tags. It will throw an error stating its looking for a string and you provided a 'dict'***.
The subnet where the EC2 instance will reside will need internet access in order for the Zui app on the local endpoint to be able to access the Zed lake on the EC2:
>#### Provide the subnet internet access
>- ```aws ec2 create-internet-gateway``` to create the IGW, record the ```InternetGatewayId```
>- ```aws ec2 attach-internet-gateway --vpc-id your-vpc-id --internet-gateway-id your-igw-id``` to attach the gateway to our ```VPC```
>- ```aws ec2 create-route-table --vpc-id your-vpc-id``` to create a route table associated with your ```VPC```. Record your ```RouteTableId```
>- ```aws ec2 create-route --route-table-id your-route-table-id --destination-cidr-block 0.0.0.0/0 --gateway-id your-igw-id``` to create the specified route:
> - ***Note: The IP/Network specified in the security group rules (x.x.x.x/32) here as well as an added security layer alongside the security group rules that are created later***
>- ```aws ec2 associate-route-table --subnet-id your-subnet-id --route-table-id your-route-table-id``` to associate the route with the subnet we created above
#### Creating the ```security group```
Now that there's a by way of the ```VPC```, a security group can be created for the ```VPC```. A ```security group``` acts like a firewall, where *ingress* and *egress* rules can be added to accept or deny network traffic.
```
aws ec2 create-security-group --group-name ZedSecGroup --description "Zed Lake Security Group" --vpc-id <your-vpc-id>
```
Record the ```GroupId```.
#### Creating the Security Group rules
Now to create the security group rules (descriptions are optional)
```
aws ec2 authorize-security-group-ingress \
--group-id your-security-group-id \
--ip-permissions 'IpProtocol=tcp,FromPort=22,ToPort=22,IpRanges=[{CidrIp=x.x.x.x/32,Description="Allow SSH access from my IP"}]'
aws ec2 authorize-security-group-ingress \
--group-id your-security-group-id \
--ip-permissions 'IpProtocol=tcp,FromPort=9867,ToPort=9867,IpRanges=[{CidrIp=x.x.x.x/32,Description="Allow Zed lake access from my IP"}]'
```
## Creating the ```EC2``` instance
>#### EC2 Instance creation
>- Choose an AMI (The OS)
>- Choose an [Instance Type](https://aws.amazon.com/ec2/instance-types/)(The machine architecture)
```
aws ec2 run-instances \
--image-id ami-03c4f11b50838ab5d \
--instance-type t3.small \
--user-data file://zed-lake.yaml \
--metadata-options "HttpEndpoint=enabled,HttpTokens=required" \
--tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=zed-lake}]' \
--network-interfaces \
'{"DeviceIndex":0,"SubnetId":"your-subnet-id","Groups":["sg-your-security-groupid"],"AssociatePublicIpAddress":true}'
```
Once the machine ready, test the SSH rule by attempting to access the instance
```
ssh zed@<ec2-instance-ip-address>
```
Also check the status of the ```zed-lake.service```
```
systemctl status zed-lake.service
```
Which should show running and enabled
And see if the EC2 is listening of the Zed lake port (9867/tcp)
```
ss -lt
```
## Connecting to the Zed Lake
Now that the actual work is over, we can open up Zui on our local machine and connect to the lake we just created.
And add some data.
And select ```Query Pool```
The Zed Lake is ready for data analysis. Consult the [docs](https://zed.brimdata.io/docs/language/overview) on how to formulate queries and transform the data. It isn't quiet ```SQL```, but if you're familiar with ```SQL``` it shouldn't take long to get comfortable with it.

View File

@@ -0,0 +1,12 @@
aws cloudformation create-stack \
--stack-name zed-lake-$(uuidgen) \
--template-body file://zed_stack.yaml \
--parameters \
ParameterKey=AmiId,ParameterValue=ami-0abcde123456 \
ParameterKey=CloudConfig,ParameterValue="$(cat ../../zed-lake.yaml | base64 -w 0)" \
ParameterKey=UniqueSuffix,ParameterValue=$(tr -dc 'a-z0-9' < /dev/urandom | head -c 10) \
--capabilities CAPABILITY_NAMED_IAM
aws cloudformation describe-stacks \
--stack-name my-zed-lake-stack-<unique-id> \
--query 'Stacks[0].Outputs'

View File

@@ -0,0 +1,259 @@
AWSTemplateFormatVersion: '2010-09-09'
Description: 'CloudFormation stack for deploying a Zed Lake.'
Metadata:
'AWS::CloudFormation::Interface':
ParameterGroups:
- Label:
default: 'Networking Configuration'
Parameters:
- VpcCidr
- SubnetCidr
- Label:
default: 'EC2 Configuration'
Parameters:
- InstanceType
- AmiId
- Label:
default: 'User Data'
Parameters:
- CloudConfig
- Label:
default: 'Unique Suffix'
Parameters:
- UniqueSuffix
Parameters:
VpcCidr:
Type: String
Default: 10.0.0.0/16
Description: CIDR block for the VPC.
SubnetCidr:
Type: String
Default: 10.0.1.0/24
Description: CIDR block for the subnet.
InstanceType:
Type: String
Default: t3.small
Description: EC2 instance type.
AmiId:
Type: AWS::EC2::Image::Id
Description: The ID of the AMI to use for the EC2 instance (e.g., latest Amazon Linux 2 or Ubuntu).
CloudConfig:
Type: String
Description: Base64-encoded cloud-config script for EC2 user data, which will handle SSH key injection.
UniqueSuffix:
Type: String
Description: User supplied unique alphanumeric suffix for resource uniqueness.
Resources:
# Networking
VPC:
Type: AWS::EC2::VPC
Properties:
CidrBlock: !Ref VpcCidr
EnableDnsSupport: 'true'
EnableDnsHostnames: 'true'
Tags:
- Key: use_case
Value: zed_lake
InternetGateway:
Type: AWS::EC2::InternetGateway
Properties:
Tags:
- Key: use_case
Value: zed_lake
AttachGateway:
Type: AWS::EC2::VPCGatewayAttachment
Properties:
VpcId: !Ref VPC
InternetGatewayId: !Ref InternetGateway
PublicSubnet:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref VPC
CidrBlock: !Ref SubnetCidr
MapPublicIpOnLaunch: 'true'
AvailabilityZone: !Select [ 0, !GetAZs ]
Tags:
- Key: use_case
Value: zed_lake
RouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref VPC
Tags:
- Key: use_case
Value: zed_lake
RouteToInternet:
Type: AWS::EC2::Route
DependsOn: AttachGateway
Properties:
RouteTableId: !Ref RouteTable
DestinationCidrBlock: 0.0.0.0/0
GatewayId: !Ref InternetGateway
SubnetRouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref PublicSubnet
RouteTableId: !Ref RouteTable
EC2SecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
VpcId: !Ref VPC
GroupDescription: Enable SSH (22) and Zed Lake (9867) access from my IP.
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 22
ToPort: 22
CidrIp: x.x.x.x/32
Description: SSH from my IP
- IpProtocol: tcp
FromPort: 9867
ToPort: 9867
CidrIp: x.x.x.x/32
Description: Zed Lake access from my IP
Tags:
- Key: use_case
Value: zed_lake
# S3 Bucket Creation
ZedLakeBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Sub 'zed-lake-${UniqueSuffix}'
VersioningConfiguration:
Status: Enabled
Tags:
- Key: use_case
Value: zed_lake
# Identity creation
ZedLakeInstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Roles:
- !Ref ZedLakeEC2Role
ZedLakeEC2Role:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- ec2.amazonaws.com
Action:
- sts:AssumeRole
Path: /
Policies:
- PolicyName: !Sub 'zed-lake-access-${UniqueSuffix}'
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- s3:GetObject
- s3:ListBucket
- s3:GetBucketLocation
Resource:
- !GetAtt ZedLakeBucket.Arn
- !Sub '${ZedLakeBucket.Arn}/*'
- Effect: Allow
Action:
- s3:ListAllMyBuckets
Resource: '*'
ZedLakeUploader:
Type: AWS::IAM::User
Properties:
UserName: !Sub 'zed-lake-uploader-${UniqueSuffix}'
Tags:
- Key: use_case
Value: zed_lake
ZedLakeUploaderPolicy:
Type: AWS::IAM::Policy
Properties:
PolicyName: !Sub 'zed-lake-uploader-${UniqueSuffix}'
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- s3:PutObject
- s3:GetObject
- s3:ListBucket
- s3:DeleteObject
Resource:
- !GetAtt ZedLakeBucket.Arn
- !Sub '${ZedLakeBucket.Arn}/*'
Users:
- !Ref ZedLakeUploader
ZedLakeUploaderAccessKey:
Type: AWS::IAM::AccessKey
Properties:
UserName: !Ref ZedLakeUploader
Status: Active
ZedLakeUploaderCreds:
Type: AWS::SecretsManager::Secret
Properties:
Name: !Sub '/zed-lake/zed-lake-uploader-${UniqueSuffix}'
Description: Credentials for the S3 Uploader IAM User.
SecretString: !Sub |
{
"ACCESS_KEY": "${ZedLakeUploaderAccessKey}",
"SECRET_KEY": "${ZedLakeUploaderAccessKey.SecretAccessKey}"
}
# EC2 Creation
EC2Instance:
Type: AWS::EC2::Instance
DependsOn:
- AttachGateway
Properties:
ImageId: !Ref AmiId
InstanceType: !Ref InstanceType
SubnetId: !Ref PublicSubnet
SecurityGroupIds:
- !Ref EC2SecurityGroup
IamInstanceProfile: !Ref ZedLakeInstanceProfile
UserData: !Ref CloudConfig
Tags:
- Key: use_case
Value: zed_lake
Outputs:
VPCId:
Description: The ID of the newly created VPC
Value: !Ref VPC
PublicSubnetId:
Description: The ID of the public subnet
Value: !Ref PublicSubnet
EC2PublicIP:
Description: Public IP address of the EC2 instance
Value: !GetAtt EC2Instance.PublicIp
ZedLakeBucket:
Description: S3 bucket for data upload
Value: !Ref ZedLakeBucket
ZedLakeUploader:
Description: Zed Lake IAM user to upload to the Zed Lake bucket
Value: !Ref ZedLakeUploader
ZedLakeUploaderCreds:
Description: Secret ARN for ZedLakeUpload identity.
Value: !Ref ZedLakeUploaderCreds

63
zed-lake/zed-lake.yaml Normal file
View File

@@ -0,0 +1,63 @@
#cloud-config
users:
- name: zed
lock_passwd: false
passwd: your-hashed-passwd # openssl passwd -6 can create a hashed password
ssh_authorized_keys:
- your-ssh-PUBLIC-KEY
shell: /bin/bash
groups: wheel
sudo: ['ALL=(ALL) NOPASSWD: ALL'] # optional, make sure to secure access to the instance/VM
write_files:
- path: /etc/ssh/sshd_config
content: |
Protocol 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
SyslogFacility AUTH
LogLevel INFO
LoginGraceTime 120
PasswordAuthentication no
PermitRootLogin no
StrictModes yes
PubkeyAuthentication yes
IgnoreRhosts yes
HostbasedAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no
PrintLastLog yes
TCPKeepAlive yes
AcceptEnv LANG LC_*
UsePAM yes
- path: /usr/lib/systemd/system/zed-lake.service
content: |
[Unit]
Description=Zed Lake Service
After=network.target
[Service]
ExecStart=/opt/Zui/resources/app.asar.unpacked/zdeps/zed serve -l :9867 -lake /home/zed/.config/Zui/lake -log.level=info -log.filemode=rotate -log.path=/home/zed/.config/Zui/logs/zlake.log
Restart=always
User=zed
[Install]
WantedBy=multi-user.target
package_update: true
package_upgrade: true
packages:
- vim
- firewalld
runcmd:
- systemctl enable --now firewalld
- firewall-cmd --permanent --add-port 9867/tcp
- firewall-cmd --reload
- dnf install -y https://github.com/brimdata/zui/releases/download/v1.18.0/Zui-1.18.0.x86_64.rpm
- mkdir -p /home/zed/.config/Zui/lake /home/zed/.config/Zui/plugins/brimcap/storage/root /home/zed/.config/Zui/logs
- chown zed:zed -R /home/zed/.config
- find /opt/Zui/resources/app.asar.unpacked/zdeps/suricata -exec chmod go+w {} \;
- /opt/Zui/resources/app.asar.unpacked/zdeps/suricata/suricataupdater
- systemctl daemon-reload
- systemctl enable --now zed-lake.service
package_reboot_if_required: true