DiaryFolio

In the ever-evolving landscape of artificial intelligence, Language Models (LMs) stand out as one of the most fascinating and impactful innovations. These LMs have revolutionized various aspects of natural language processing, enabling machines to comprehend and generate human-like text with astonishing accuracy. In this blog post, we'll embark on a journey to demystify LMs, exploring key terminologies and shedding light on their inner workings. The below blog will put a summary Understanding Key Terminologies: 1. Tensors Tensors are fundamental data structures used in deep learning frameworks like TensorFlow and PyTorch. They are multi-dimensional arrays that allow efficient representation of complex data, such as images, text, and numerical data. In the context of LMs, tensors serve as the primary means of storing and manipulating input data, facilitating the training and inference processes. 2. Quantization: Quantization is a technique used to reduce the memory and computation...

 Kuberenetes has given a 'Software view' for the 'hardware' world. That too all resources consumed via modern definitions using json/yaml and via API.  Kuberentes segments the compute resources into Worker nodes & Master Node(s) and contain persistent entities called 'Kubernetes Objects' including Containerised applications Cluster and Associated nodes Resources to these nodes The policies and tolerances on how the applications interact and behave Below is a good diagram of the various components Each component can be defined by software/code and scalable which makes kubernetes the de-facto building framework for modern micro-service applications.  In most of the scenarios the components can be tiered into  Host/Virtual machines Kuberentes Platform Containers Microservices It is hence very important to understand the difference between traditional 2 tier model and kuberentes 4 tier model for all your Operational, Security and Observability needs for a succes...

 Google Domains & Setting up customised email Summary Google services are extremely powerful and can be customised to great extend than you could ever imagine. In this article, we will look into setting up SMTP & Email forwarding from your own domain and gmail at zero cost (Well cost of the domain) Pre-Reqs A domain and Ability to amend DNS MX records (Example uses google domains and addresses are availble for around $10 per year) Gmail address Steps Domain Steps - Setting up Email Forwarding Setup Email Forwarding Navigate to:  https://domains.google.com/registrar/<your_domain>/dns Select  Email -> Email Forwarding -> Add email alias Add -> Verify You should get an email to your "@gmail.com" address  Verify it by clicking on the link Verify if the MX records have come up in your DNS URL Select DNS and Verify MX record has been added Please wait some time (an hour or so) and send an email to your new address  " admin@diaryfolio.com " and ...

  Deploying awx into kubernetes Intro AWX provides a web-based user interface, REST API, and task engine built on top of Ansible. This article will summarise steps of installing AWX into Kubernetes using Operator Pre-Reqs Familiarity with Kubernetes (k8s) Bit powerful node machine of 4vCPU & 8GB RAM AWX is exposed on port 9080, so enable firewall accordingly (As default port 80 will have collision mostly in a Kubernetes environment) Build Operator from code (Optional Step) Unfortunately at the time of writing, the official repository doesn't give a operator yaml directly, but suggests to build from code. But we use a pre-built operator yaml and hence below step of creating from source-code is optional apiVersion: v1 kind: PersistentVolumeClaim metadata: creationTimestamp: null labels: io.kompose.service: n8n-claim0 name: n8n-claim0 spec: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi status: {} Package components The deployment i...

  Kubernetes & n8n: Setup n8n using K8S (Part 2) Summary We have briefly discussed on automating various tasks using n8n and installation of n8n within Kubernetes in Part1 . In this part, we mostly concentrate on how to enable TLS (for https) and terminate before it hits n8n Pre-Reqs Setting up of n8n in Kubernetes ; Read  Part1 Knowledge of TLS, certificates Summary Steps Ensure n8n is configured properly with by http Implement Kubernetes Ingress with https There are two options from here End to end TLS by redirecting proxy to n8n  TLS termination at proxy and private network to be non-secure We follow the second option as it is easier, thus creating Ingress and pointing to n8n service Steps Steps in  lab_server Ensure Certificate is created and implemented as a secret in Kubernetes preferably in same namespace kubectl -n n8n create secret tls tls-secret --key test.key --cert test.crt Use the same tls-secret in the Ingress config apiVersion: networking.k8s.io...

Setting up Kubernetes Cluster in your home lab Summary Kubernetes is the future of automating deployment, scaling, and management of containerised applications which makes development of projects easier and portable. Additionally setting up such an environment within your local lab environment makes it easy to deploy and test out softwares at unprecedented pace. You might have already setup Kubernetes in your laptop and must have used minikube to do so. But what if you want to setup it in your remote home lab? For instance i've got various DELL server ( lab_server ) running in my home lab and don't want my laptop to bear the pain of Kubernetes, but outsource the workloads to my  lab_server  while administering from laptop k3s is a lightweight alternative and k3sup will automate such an installation into your  lab_server  and pair your laptop with it Pre-Reqs Understanding of Kubernetes sudo permission on your home lab and connectivity from your la...

 AWS Lambda: Function to stream logs via SQS Summary As part of logging and monitoring strategy it is quite important to emit data from AWS services to another service or system or aggregation layer. AWS SQS (Amazon Simple Queue Service) is a great tool to communicate between such micro-services with real-time & between software components at any volume. Aim of this article is a snippet to stream logs via SQS to an external service (like logstash or siem tools) using Lambda Function Pre-Reqs Permission to pull data from specific SQS queue Steps Ensure lambda function can reach the SQS queue Below is a snippet of code to push the data in real-time import gzip import json import base64 import boto3 import time def lambda_handler(event, context): sqs = boto3.client('sqs') account = boto3.client('sts').get_caller_identity()['Account'] queue_url = "https://sqs.eu-west-1.amazonaws.com/12345567928/my-app-{}.fifo".format(account) cw_da...