Skip to main content

DevOps : Search and replace files with JSON dataset and template Engine

Today's code build & continuous deployment models are highly diverse thus leading to handwritten and complicated perl/awk/sed scripts. DevOps should come out of age old hand-crafted find and replace scripts with much modern template engines.
              Of course template engines are available in wide variety. All enterprise configuration management (chef, puppet, ansible ) software are equipped with their own flavour of template engines and playbooks.  This article however concentrate on "Mustache" template  which is logicless template system and work on any text based data (Web pages, scripts, dataset, config files etc..)
              The example below focusses on replacing dynamic text using a JSON dataset. Let's define terminologies
  • Source :   Template Parent Directory with all files/directories +  dynamic variables in it
  • Dataset:    JSON based self defining dataset to replace the above source(s)
  • Params:    Extra parameters that are supplied (eg: Environment PROD, DEV etc..)
The key idea here is to replace all templates (recursively) within the "Parent Source Directory" and find and replace with dataset. In summary this is what I'm trying to achieve
  • All template files in "Source" to have an extension of  ".mustache"
  • Recursively scan whole of "Source" directory and children and identify  .mustache files
  • Feed these files into mustache renderer
  • Rending information supplied using "Dataset"
  • Filter out required elements of "Dataset" using input "Params"
  • Copy directory structure and files into a temporary location with filled values



Popular posts from this blog

Create your own Passport Photo using GIMP

This tutorial is for semi-techies who knows a bit of GIMP (image editing).   This tutorial is for UK style passport photo ( 45mm x 35 mm ) which is widely used in UK, Australia, New Zealand, India etc.  This is a quick and easy process and one can create Passport photos at home If you are non-technical, use this link   .  If you want to create United States (USA) Passport photo or Overseas Citizen of India (OCI) photo, please follow this link How to Make your own Passport Photo - Prerequisite GIMP - One of the best image editing tools and its completely Free USB stick or any memory device to store and take to nearby shop A quality Digital camera Local Shops where you can print. Normally it costs (£0.15 or 25 US cents) to print 8 photos Steps (Video Tutorial attached blow of this page) Ask one of your colleague to take a photo  of you with a light background. Further details of how to take a photo  yourself       Take multiple pictures so that you can choose from th

Elastic Beats on pfSense : Installation and configuration

Summary Though in many cases syslog is preferred to transport the pfSense logs to external system, Elastic beats provides quite a niche way to send the logs while modelling the data alongside. This makes it ready-made to send to ElasticSearch directly and get ready-made outcomes like SIEM, performance etc. Pre-reqs A build server (preferably Ubuntu or Fedora) with internet connectivity shell access to pfsense server Basic knowledge of Elastic Stack (filebeat.yml configurations etc) Ensure connectivity is allowed from pfsense machine to your Elastic Stack receiver Setup Summary Connectivity tests Install dependencies in build server (vagrant, virtualbox, gmake, go etc) Download Elastic Beats source Make elastic Beats package for FreeBSD Copy binary packages to pfsense server Configure Beats to send to destination Configure ElasticSearch to view the data Installation Steps Connectivity tests Logon to pfsense server via Shel

Syslog Standards: A simple Comparison between RFC3164 & RFC5424

Syslog Standards: A simple Comparison between RFC3164 (old format) & RFC5424 (new format) Though syslog standards have been for quite long time, lot of people still doesn't understand the formats in detail. The original standard document is quite lengthy to read and purpose of this article is to explain with examples Some of things you might need to understand The RFC standards can be used in any syslog daemon (syslog-ng, rsyslog etc.) Always try to capture the data in these standards. Especially when you have log aggregation like Splunk or Elastic, these templates are built-in which makes your life simple. Syslog can work with both UDP & TCP  Link to the documents the original BSD format ( RFC3164 ) the “new” format ( RFC5424 ) RFC3164 (the old format) RFC3164 originated from combining multiple implementations (Year 2001)