Kubernetes init database in a dirty way

Sometimes you need a dirty trick initialized the kubernetes running database with a set of sql statements (ddls). Here you are a simple example.

init-db:

kubectl exec -i --namespace timescaledb svc/csdb -c timescaledb -- \

        psql -U postgres -d cas --file=- <ddls/00-init-database.sql;

init-db-tables:

@for f in $(shell ls ddls/$(FILE_PREFIX)*.sql | grep -v 00-init); do \

kubectl exec -i --namespace timescaledb svc/csdb -c timescaledb -- \

            psql -U postgres -d cas --file=- <$$f; \

done

But what if you would like to variables in your script.

Here's a solution:

bash -c "cat <<EOF

$(< ddls/00-init-database.sql_tpl)

EOF" | kubectl exec -i --tty --namespace timescaledb ${PGMASTERPOD} -c timescaledb -- \

    psql -U postgres -d ${DBNAME}

Maven - Debugging the PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

From time to time that infamous exception attacks us. Usually, it is just as simple as import the valid key to your cacerts trustStore or specify the right path. But what if that does not help this time? After double check everything seems to be right. It's time to dive deeper. 

That simple debug steps make your live longer:

 

1. check the actual settings were applied: `-XshowSettings:properties` 
2. If still no idea what's wrong, try the ultimate weapon: `--Djavax.net.debug=all` - and be prepared for long log file ;). 

And couple more takeaways: 

1. The `.mavenrc` overrides whatever comes even in MAVEN_OPTS passed from environment.
2. The `${USER}` variable is not evaluated when put into `${PRJECT_DIR}/.mvn/jvm.config` file.

RabbitMQ - python publish to the channel required mTLS authentication

To make possible publish an event to the RabbitMQ channel secured with TLS certificate one requires delivering the `ssl_options` whilst creating a connection. The flag `ssl.CERT_REQUIRED` has to set for the `verify_mode` parameter of the default security context. Additional the `verify_flags` has to be set to `VERIFY_X509_TRUSTED_FIRST`. Moreover the cert_chain has to be populated with the valid certificate issued by CA specified when creating security context object. # Imports

import pika
import ssl

# Create method for obtaining connection

rabbit_url="170-187-131-61.ip.linodeusercontent.com"
path_to_cert=""../infra/secrets/rabbitmq/tls.crt"
path_to_key="../infra/secrets/rabbitmq/tls.key"
path_to_cafile="../infra/secrets/rabbitmq/ca.crt"
rabbituser, rabbitpass = 'rabbituser', 'rabbitp@ss'


def get_connection(kind="tls"):
   if kind == "tls":
        context = ssl.create_default_context(cafile=path_to_cafile)
        context.verify_mode = ssl.CERT_REQUIRED
        context.verify_flags = ssl.VERIFY_X509_TRUSTED_FIRST
        context.load_cert_chain(path_to_cert, path_to_key)
        ssl_options = pika.SSLOptions(context, rabbit_url)
        credentials = pika.PlainCredentials(rabbituser, rabbitpass)
        parameters = pika.ConnectionParameters(host=rabbit_url, 
                                               port=5671, 
                                               ssl_options=ssl_options,
                                               credentials=credentials)
        return pika.BlockingConnection(parameters)
    else:
        credentials = pika.PlainCredentials(rabbituser, rabbitpass)
        parameters = pika.ConnectionParameters('localhost', 5672, '', credentials)
        return pika.BlockingConnection(parameters)

# Use the connection and publish event

    logging.info(f"Processing command: {cmd}")
    with get_connection() as connection:
        with connection.channel() as channel:
          channel.queue_declare(queue='_events', durable=True)
          channel.basic_publish(
              exchange='',
              routing_key='_events',
              body=cmd,
              properties=pika.BasicProperties(
                  delivery_mode=2,  # make message persistent
              ))
          return " [x] Sent: %s" % cmd

SQL window functions by example

The following query computes number of metrics written in a given hour

select time, workflow, metrics_written, 
       CAST(TO_CHAR(time, 'YYMMddHH24') as BIGINT) as bin, 
       min(metrics_written) over w, max(metrics_written) over w
  from internal_write
 where workflow = 'workflow_name'
   and output = 'sql'
 window w as (PARTITION BY CAST(TO_CHAR(time, 'YYMMddHH24') as BIGINT) 
                  ORDER BY time DESC RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)
  order by time desc

This one in turn shows number of metrics written within passing 5 minutes window

select time, metrics_written, 
       max(metrics_written) over w - min(metrics_written) over w as metrics_written_within_5_minutes
  from internal_write
 where workflow = 'workflow_name'
   and output = 'sql'
window w as (order by time range (interval '5 min') preceding)
 order by time desc
 

How to intercept requests with lua script in Istio service mesh

Add custom header to the responses comming out the given Kubernetes application

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: add-custom-header-envoyfilter
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      app: httpd-proxy
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
        # portNumber: 80
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
            subFilter:
              name: envoy.filters.http.router
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.filters.http.lua
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua"
          inlineCode: |
            function envoy_on_response(response_handle)
              response_handle:logDebug("Adding custom header to the response")
              response_handle:headers():add("X-Custom-Header", "1.2.3.4")
            end

The above works on the response. To modify the Lua `inlineCode` should look like this

  function envoy_on_request(request_handle)
  	request_handle:logInfo("Received request for " .. request_handle:headers():get(":path"))
  end

Note that by default the istio logging level is set to warning. You need to turn loggin to info to make it appear in the log file.

Oneliner for generating a self-signed x.509 certificate for scripting purposes

Purpose

There are cases when we'd need a quick way to generate x.509 certificate for scripting adhoc purposes without a need to create a CSR file or putting the values from the terminal by hand.

Prerequisites

Tested with Centos 7/stream8. $yum install openssl -y

Solution

openssl req -subj '/CN=lol.mazia.rz/O=MZIARZ/C=US' -new -newkey rsa:2048 -sha256 -days 365 -nodes -x509 \
      -keyout /etc/httpd/certs/private-key.pem \
      -out /etc/httpd/certs/cert.pem

And here you go. Have fun.

Pandas write integer dataframe column as a timestamp into the database

To display DataFrame's datatypes use dtypes property.

> print(df.dtypes)

last_login    float64
created_at      int64

In order to cast the `float64` or `int64` column into the `datetime64` one the to_datetime method has to be used. Note that specifying the time unit may be needed to make the conversion correct. Assuming our values contain a UNIX timestamps in seconds the `unit="s"` extra parameter will be required.

df['last_login'] = pd.to_datetime(df['last_login'], unit="s")
df['created_at'] = pd.to_datetime(df['created_at'], unit="s")

Finnaly we receive the following types:

> print(df.dtypes)

last_login    datetime64[ns]
created_at    datetime64[ns]

Now the timestamps in the database will be valid instead of '0000-00-00 00:00:00' we spotted without valid conversion.

Terraform, DigitalOcean, Failed to query available provider packages

There is slight chance that during getting started with Digital Ocean tutorial you encountered the following error

$ terraform init                

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/digitalocean...
╷
│ Error: Failed to query available provider packages

The solution to that is deliver required_providers section into your main.tf

terraform {
    required_providers {
        digitalocean = {
            source = "digitalocean/digitalocean"
            version = ">= 2.4.0"
        }
        kubernetes = {
            source = "hashicorp/kubernetes"
            version = ">= 2.0.0"
        }
    }
}

Now the terraform init command should be executed successfully

Minikube - Failed to pull image - connection refused

Failed to pull image "nginx": rpc error: code = Unknown desc = Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on 192.168.64.1:53: read udp 192.168.64.5:51079->192.168.64.1:53: read: connection refused

Below yuo can find a sequence of commands that worked for me. First go to minicube shell and try to re the registry-1.docker.io

$ minikube ssh
$ nslookup registry-1.docker.io

If it does not show the list of ips we've got the culprit.

$ su - root
# vi /etc/resolv.conf
# put a your favourite public DNS instead the one its defined

One extra thing you need to do is stop the systemd-resolved process

# systemctl stop systemd-resolved

Now there should fine, kuberetes will be able to download images.

How to setup php.ini for PHP develpement server

Php development server is excellent thing for quick development. But seems not very clear how to setup custom php parameters such as upload_max_filesize. Putting php.ini into main folder does not work for me, and even setting with ini_set failed. Desperatly I desided to read manual. Yes, sometimes it is worth do so ;>. But

/usr/local/Cellar/php/7.4.8/bin/php -S localhost:8081 -d upload_max_filesize=12M

And remember phpinfo() function is your friend. alternatively you can always set global value in the vi /usr/local/etc/php/7.4/php.ini file.

Wiremock standalone - SLF4J: Failed to load logger implementation

Seems like `wiremock-standlone` comes without SLF4J implementation (?).

$ java -wiremock-standalone-2.23.2.jar com.github.tomakehurst.wiremock.standalone.WireMockServerRunner --proxy-all="http://localhost:8090" --verbose
2019-07-15 11:41:35.091 Verbose logging enabled
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
2019-07-15 11:41:35.650 Verbose logging enabled

Let's fix it by adding simple implementation on the classpath:

$ java -cp ~/.m2/repository/org/slf4j/slf4j-simple/1.7.12/slf4j-simple-1.7.12.jar:wiremock-standalone-2.23.2.jar \
  com.github.tomakehurst.wiremock.standalone.WireMockServerRunner \
  --proxy-all="http://localhost:8090" \
  --port 0 \
  --verbose \
  --print-all-network-traffic

How to install SSL certificate in java keystore

The Problem:

After adding maven repository secured with https protocol the following error appeared:

sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.
certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

But how to get the certificate and make java can find it? Step 1. Fetching the certificate

echo | openssl s_client -showcerts -connect nexus:443 2>/dev/null | \
 awk '/-----BEGIN CERTIFICATE-----/, /-----END CERTIFICATE-----/' > nexus.crt

Step 2. Find out where the java keystore is located The following command will show java home dir

$ /usr/libexec/java_home
/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home

The keystore is jre/lib/security/cacerts file. Step 3. Install the certificate with keytool command

sudo keytool -importcert -alias nexus01 -keystore /Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/security/cacerts -file nexus.crt 

Hint: use changeit or changeme as a default password. Step 4. Verify your certificate is on the list

$ keytool -list -keystore /Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/security/cacerts | grep nexus01
nexus01, Nov 15, 2018, trustedCertEntry, 

Spring MVC - Redirect 404 when Optional.empty returned

Optional returned from controllers method

If you need to show 404 error page when no object found in your backend you can think of returning Optional from the controller's method and may expect Spring MVC takes care of exposing the value if present and throws 404 if empty.

Solution

The mechanism of controller advice comes to play. We can create a class that will be used to modify controller responses.

There are actually two features used:

• @ExceptionHandler which is used for translation exceptions into 404 error response
• ResponseBodyAdvice interface which cause body will be processed before been returned from controller endpoint.

Here you are piece of code that should gather above mentioned into one universal class

@ControllerAdvice
public class OptionalResponseControllerAdvice implements ResponseBodyAdvice {

 @Override
 public boolean supports(MethodParameter returnType, Class converterType) {
  return returnType.getParameterType().equals(Optional.class);
 }

 @Override
 public Object beforeBodyWrite(Object body, MethodParameter returnType, MediaType selectedContentType,
   Class selectedConverterType, ServerHttpRequest request, ServerHttpResponse response) {
  if (returnType.getParameterType().equals(Optional.class)) {
   return ((Optional<?>) body).orElseThrow(() -> new NotFoundException("No object found: " + request.getURI()));
  }
  return body;
 }

 @ExceptionHandler(UnauthorizedException.class)
 public ResponseEntity<Map<String, Serializable>> handle(NotFoundException e) {
  return ResponseEntity.status(HttpStatus.NOT_FOUND).body(e.getMessage());
 }

}

Angular 5 and underscore library

Are you missing the underscore library in your most recent Angular 5 project?

It is easier than you may think - the only you need to do is just add two dependencies to your package.json file.

Step 1. Install dependencies

You can do this with npm command.

npm install --save @types/underscore

and next

npm install --save underscore

Step 2. Use it

In order to use underscore library in your component or service class you have to import library in the following way:

import * as _ from "underscore";

That is basically all - now you can take all the goodies from underscore. For example, to fetch all unique key values from an array of key-value struct we can do this way:

public uniqueKeys(input: {[key: string]: string}[]): string[] {

   let keys = _
     .chain(input)
     .map(a => _.keys(a))
     .flatten()
     .uniq(false)
     .value()

HttpClient - ssl - unable to find valid certification path to requested target

The javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target usually stands for issues with accepting/validating SSL/TLS certificate given by web server. ClientHandshaker class is responsible for handshaking from the client side. It shares common logic such control flow and key generation with its counterpart ServerHandshaker though common parent class called Handshaker.

void processMessage(byte type, int messageLen) throws IOException {
 ...
 switch (type) {
 ...
 case HandshakeMessage.ht_certificate:
       if (keyExchange == K_DH_ANON || keyExchange == K_ECDH_ANON
                    || keyExchange == K_KRB5 || keyExchange == K_KRB5_EXPORT) {
                fatalSE(Alerts.alert_unexpected_message,
                    "unexpected server cert chain");
                // NOTREACHED
            }
            this.serverCertificate(new CertificateMsg(input));
            serverKey =
                session.getPeerCertificates()[0].getPublicKey();
            break;

Let's go briefly to the details 1. Creating CerificateMsg just takes input as a HandshakeInStream, reads first 24 bytes as a chainLenght, so can read the cert as a byte array and finally instantiates X.509 certificate, as below

cf = CertificateFactory.getInstance("X.509");
cf.generateCertificate(new ByteArrayInputStream(cert)

2. serverCertificate method will deletegate cerificate validation to the Trust Manager taken from sslContext. To solve the error the sslcontext have to be feeded with TrustStrategy that accepts self-sighed cerificates as trusted. Apache Http Client comes with org.apache.http.conn.ssl.TrustSelfSignedStrategy to do so.

HttpClients.custom()
                .setSSLSocketFactory(new SSLConnectionSocketFactory(SSLContextBuilder.create()
      .loadTrustMaterial(TrustSelfSignedStrategy.INSTANCE)
      .build(), NoopHostnameVerifier.INSTANCE))
                .build();

Java - how to encode url right

Surprisingly its not an easy thing to encode URL once one wanted to make a request with Apache HttpClient. since most obvious URLEncode seems does not work well, though. URI baseUri = new URI("http://localhost:8090"); baseUri.getScheme(); URI path = new URI("http", null, "localhost", 8090, url, null, null);

Bash - how to do string split

STR=A.B.C && STR2=${STR%.*} && echo ${STR2%.*} && echo ${STR2#*.} && STR4=${STR#*.} && echo ${STR4#*.}

DELIMITER='.' && STR=A.B.C && echo ${STR%${DELIMITER}*}

for i in $(echo A.B.C | sed -e 's/\./ /g'); do echo $i; done

bash - how to check if string starts with specific prefix

Let's assume we want to process only files that start with specific string (eg. 2018). Our script is listening to every new file appeared in given directory, so let's assume we cannot do this at the os level directly. Our script is invoking the following command:

$ convert.sh /path/to/2018_file.pdf 

To process all files begins with 2018 we can do this way:

for i in `ls`; do [[ $i == 2018* ]] && echo $i; done

Or, if we'd like to process all but 2018 files we can do this way:

for i in `ls`; do [[ $i != 2018* ]] && echo $i; done

Or even more general with help of regexp we can do

for i in `ls`; do [[ $i =~ ^2018 ]] && echo $i; done

Happy New Year!!

bash - how to concatenate array with given delimiter

Let's declare array

 a=(aa bb cc) 

Now we can display the content with

$echo ${a[@]}
aa bb cc

First, give some naive solutions a try

$for el in ${a[@]}; do echo -n ,$el; done
,aa,bb,cc

$printf ",%s" ${a[@]}
,aa,bb,cc

Both work well, right? Hmm.. almost.

The last thing to do is get rid of the first comma

$for ab in ${a[@]}; do echo -n ,${ab}; done | cut -c2-
aa,bb,cc

Why Spring returns 404 for my request?

When Spring Web MVC returns 404 in response to our request it means that cannot find handler for request "signature". As a signature we mean request mapping and request parameters.

---

How Spring resolves handler for a given url address?

The class responsible for matching mappings with requests is org.springframework.web.servlet.DispatcherServlet. The specific handler will be determined by applying handler mappings. Method responsible for that is called getHandler and it takes HTTP servlet request (Good old javax.servlet.http.HttpServletRequest instance). It basically goes through all registered handler mappings and tests each in order. The first match win. As the result, it returns an object of HandlerExecutionChain class.

AbstractHandlerMethodMapping.java

protected HandlerMethod lookupHandlerMethod(String lookupPath, HttpServletRequest request) throws Exception {..}

---

Resolving details

At first, it looks for a direct match. eg /api/logs/3/4 Next, it starts to look for matching patterned paths eg. /api/logs/{parent}/{child}. Takes all matched and is looking for the best match.