Kotlin scripting

Jun 8, 2024 | 7 minutes | Translation: ru (Пишем скрипты на Котлине)

I recently wrote CI Job. Initially, I wanted everything in Python in one file and without library dependencies. It turned out to be about 600 lines. Clearly too much for this approach. So it clearly needs to be refactored.

I decided not to continue in Python:

weak typing leads to more complex debugging
curve / complex work with lists

It feels like this right now:

BASH for one-liners and simple scripts up to 10-20 lines
Python without libraries – for larger scripts (up to 200 lines) and when specific libraries are needed (AI, for example)
Kotlin for longer scripts and cli (I used to consider Golang for cli, but it has its own problems, and Kotlin has moved in this direction)

Scripts in Kotlin

For the second version, I take Kotlin in script mode. 2 libraries:

Kotlin Serialization – a modern script without JSON is unlikely to be
[Klickt](https://ajalt.github.io/clikt /) – it is necessary to process command-line arguments, there is nothing at all in the Kotlin itself

Briefly how to use Kotlin for scripts:

Create a file with the main extension.kts and launch: kotlin my-script.main.kts
You can use shebang: #!/usr/bin/env kotlin
Multiple files can be used: @file:Import("utils.main.kts")
You can download dependencies: @file:DependsOn("com.google.code.gson:gson:2.11.0")
The compiler caches dependencies and script build results
There is an auxiliary library for working with script files – kotlin-shell
You can also write tests
All this can be put in the Docker
If you want more, there is Kscript

It turned out that Kotlin Serialization requires a compiler plugin with the full path to the corresponding jar (is part of the compiler), and @file:CompilerOptions("-Xplugin=${KOTLIN_HOME}/lib/kotlinx-serialization-compiler-plugin.jar")it doesn’t work.

It turned out to be such a creak of simple work with JSON:

#!/bin/sh
///bin/echo >/dev/null <<EOC
/*
EOC
kotlinc -script -Xplugin="${KOTLIN_HOME}/lib/kotlinx-serialization-compiler-plugin.jar" -- "$0" "$@"
exit $?
*/
@file:DependsOn("org.jetbrains.kotlinx:kotlinx-serialization-json:1.7.0")
@file:DependsOn("com.github.ajalt.clikt:clikt:4.4.0")

import kotlinx.serialization.Serializable
import kotlinx.serialization.json.Json
import kotlinx.serialization.encodeToString

@Serializable
data class Data(val a: Int, val b: String)

val json = Json.encodeToString(Data(42, "str"))
val obj = Json.decodeFromString<Data>(json)

println(json)
println(obj)

At the beginning, there is a bit of “black magic”: the same file is first interpreted as a shell script (and sets the path taking into account the environment variable), and then as a Kotlin file.

Next, we import dependencies and work with JSON.

And here is a Dockerfile:

FROM eclipse-temurin:21-jre

# Install dependencies for the script (jc, git), unzip for compiler unpack,
# and jq just in case it will be useful
RUN apt-get update \
&& apt-get -y install jc jq git unzip \
&& rm -rf /var/lib/apt/lists/* \
&& mkdir /app

# Install Kotlin compiler
ARG KOTLINC_VERSION="2.0.0"
RUN wget -q "https://github.com/JetBrains/kotlin/releases/download/v$KOTLINC_VERSION/kotlin-compiler-$KOTLINC_VERSION.zip" && \
    unzip "kotlin-compiler-$KOTLINC_VERSION.zip" -d /usr/lib && \
    rm "kotlin-compiler-$KOTLINC_VERSION.zip"

ENV PATH $PATH:/usr/lib/kotlinc/bin:/app
WORKDIR /app

COPY json.main.kts ./

# Cache dependencies and compilation result for better start-up speed
ENV KOTLIN_MAIN_KTS_COMPILED_SCRIPTS_CACHE_DIR /app
RUN /app/json.main.kts && find /app/.m2 -type f -exec chmod 644 {} \;

ENTRYPOINT ["/app/json.main.kts"]

Of the pros, we explicitly set the JRE and Kotlin versions, and of the cons, it is quite large for a simple script. I did not find ready-made images for scripts.

Kscript

Because of these 2 points, I decided to watch the Script. The docker is cleaner there:

FROM kscripting/kscript
ENV KOTLIN_HOME /opt/kotlinc

RUN mkdir /app
ENV PATH $PATH:/app
WORKDIR /app

COPY json.kts ./

# Cache dependencies and compilation result for better start-up speed
ENV KOTLIN_MAIN_KTS_COMPILED_SCRIPTS_CACHE_DIR /app
RUN /app/json.kts

ENTRYPOINT ["json.kts"]

The script itself is very similar:

#!/usr/bin/env kscript
@file:CompilerOptions("-Xplugin=${KOTLIN_HOME}/lib/kotlinx-serialization-compiler-plugin.jar")
@file:DependsOn("org.jetbrains.kotlinx:kotlinx-serialization-json:1.4.1")
@file:DependsOn("com.github.ajalt.clikt:clikt:4.4.0")

import kotlinx.serialization.Serializable
import kotlinx.serialization.json.Json
import kotlinx.serialization.encodeToString

@Serializable
data class Data(val a: Int, val b: String)

val json = Json.encodeToString(Data(42, "str"))
val obj = Json.decodeFromString<Data>(json)

println(json)
println(obj)

The docker image of cscript is outdated: kotlin 1.7 is there. And kotlin-serialization-json depends on the compiler – you need to select the version. Cliket also depends on the Kotlin version and you also need to select an older version.

This is where the introduction to Jscript ends. Let’s go back to the standard scripts.

Simplify the work

Then I decided to make a basic image so that the Dockerfile script looks something like this:

FROM stepin/kotlin-scripting

COPY json.main.kts ./

# Cache dependencies and compilation result for better start-up speed
RUN /app/json.main.kts

ENTRYPOINT ["/app/json.main.kts"]

Yeah, it works :).

You can also run it without creating a Dockerfile:

# run inline script
docker run --rm -i stepin/kotlin-scripting 'println("Hello, world!")'

# run script from file
docker run --rm -i stepin/kotlin-scripting - < script.main.kts

# map current folder into container if it's needed
docker run --rm -i -v "$(PWD):/data" -v "$(HOME)/scripts/my.main.kts:/app/my.main.kts" -w /data --user "$(id -u)" stepin/kotlin-scripting

We collect it in a local file

As a bonus, let’s see how the script can be packaged into a single file with all the dependencies, and even compiled into a native binary.

There are 2 options:

build in JVM, then Graal converts to native code
Kotlin/Native immediately works with native code

Fat jar

It can be assembled using Gradle. Or via Cscript. There is no easy way through kotlin/kotlinc.

Kotlin/Native

Let’s try Kotlin/Native, for this you need to install a special compiler separately (https://github.com/JetBrains/kotlin/releases/tag/v2.0.0 kotlin-native-prebuilt- )– either from releases from github, or like this:

brew install --cask kotlin-native

And then we build:

kotlinc-native 1.main.kts -o 1 -language-version=1.9

But this does not work on version 2.0.0 (even with the -language-version=1.9 option). But it works for regular kt files.:

fun main() {
    println("Hello, World!")
}

And build:

kotlinc-native hello.kt -o hello -opt
chmod +x hello.kexe
mv hello.kexe hello

It turns out 2 files:

- hello.exe on 499K – the actual script
- hello.exe.dSYM on 96B – optional debugging information (folder)

It starts quickly:

$ time ./hello
Hello, World!
./hello  0,00s user 0,00s system 70% cpu 0,008 total

So either look for some previous version that works with scripts. Or wait for a new one. I’ll wait, I don’t really need native CLIs right now.

I note that the binary contains some dependencies:

$ file hello
hello: Mach-O 64-bit executable arm64

$ otool -L hello
hello:
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1345.100.2)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1700.255.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 2420.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 2420.0.0)

Roughly speaking, it looks like a C++ application, and not like Golang with no dependencies at all. Therefore, FROM scratch will not work, but it is usually not necessary. In any case, this is some kind of Linux base image (60 megabytes), not a 600 JVM.

Native build via Graalvm

We collect the fat jar (the example above), then https://www.graalvm.org/22.0/examples/java-kotlin-aot / .

I used Graalvm before, I didn’t really like it: the computer is doing something with all its might, for a long time and not every time successfully.

The subtotal

When you need a native application and there are Java dependencies, we use Graalvm. When there are no Java dependencies, Kotlin/Native builds faster.

Of course, the native cli is still too big (from a perfectionist point of view) – after all, hello world is less than 1 kb (or so), and not 500 times more (using the example of Kotlin/Native). Golang is better here (it is faster to assemble in the case of Graalvm and less binary), but the code itself is easier and more readable to write in Kotlin.

So let’s hope that the compiler will also actively develop in the native direction. Although there are more problems with second-class script support than with size.

Replacing awk

Another bonus. Go straight to the code:

kscript -t 'lines.split().select(10,1,12).print()' some_flights.tsv

Or on a clean Kotlin:

kotlin -e 'generateSequence(::readlnOrNull).forEach{ val f = it.split("\t"); println("${f[2]}\t${f[1]}") }' -- - < example.tsv

Very convenient. Otherwise awk is too strange, but at the same time it is needed from time to time.

More examples are here: https://holgerbrandl.github.io/kotlin/2017/05/08/kscript_as_awk_substitute.html

Сode

Sources from this article is at https://github.com/stepin/kotlin-scripting .