Kotlin Multiplatform with Compose - Dam Monitor

Lets actually do something! - changing direction, Ktor and kotlinx.serialization save us

Apr 25, 2024

So I’ve been using my brother’s desire for an app to notify him if a particular dam has changed much in level as a learning tool for Kotlin Multiplatform with Compose for Andoid/ iOS / Desktop. Now I have got the basics of testing and running the app working in Android and Desktop, will do the iOS later - but in the interest of actually getting something done I will first get a working version of the app going.

So looking at our inbox:

# Inbox
- [x] read depth from page
  - [x] Create the page from file
  - [x] Create the page from url
  - [x] Read things from the page
  - [x] Get DamReading from the page
  - [x] No depth in page gives no DamReading
  - [x] Get date of DamReading
  - [x] Get depth from DamReading
- [ ] decide whether depth is changed
- [ ] notify user if depth is changed
- [ ] allow use****r to change parameters
- [ ] notify user if depth cannot be read
- [x] load document from text file
- [x] load document from page
  - [x] what does it do with a bad url? 
- [x] get title of page from document
- [x] get table by class from document
- [x] get a dam reading from the table
- [x] get latest dam reading from table of many when latest is first
- [ ] check if a date is before or after another date
- [ ] check the time between two dates
- [ ] ui?

So what do we need to do next? We really need to decide if the depth has changed. We can get the depth from the page, so deciding if the depth has changed is next. What does that break down into though? Lets try this:


- [ ] decide whether depth is changed
  - [x] load last depth notified from config
  - [ ] load change in depth to notify from config
  - [ ] read current depth
  - [ ] compute abs(last depth - current depth)
  - [ ] compare to change in depth to notify

This is always kind of how I thought of things but “Getting Things Done - The Art of Stress-free Productivity” gave me a method of looking at it more methodically.

So I see 5 simple things that will lead to tests and then to functionality. Lets go!

The first one was already done, so the load change in depth to notify from config is the same process as load last depth notified from config. I just need a config variable that is a double, and I need implementations (and therefore tests) in desktop and android. and eventually iOS. It occurred to me as I did it, maybe I should be writing one test in desktop and making it pass for the DesktopConfigProvider, one for AndroidConfigProvider, then one for ConfigProvider and make the other two override it at that point. Would have been smaller steps. Do I care? Not really, no fear about that larger step as

J. B. Rainsberger

says.

I had added a load web address from config, then took it back out. I don’t need it in this version, it is just added complexity. Read current depth is sort of done too. I don’t have a test that gets it directly from the webpage, I have one that gets it from a stored page, and I have one that checks to see that I can in fact get a web page document. So the most I could additionally do is see that I do get a depth that is a valid double but not check its value. There is probably value in that test as it will tell me if they change their format.

So next question is all the tests I’ve written on the web page parsing involve either just the getting of the document, or use the static saved web-page. So lets write a little test to make sure we can actually get the depth from the live web-page. Obviously this can’t easily check that the depth is correct but it can check that it is sensible (non-negative non-null non-infinite). So here is our test:

    @Test
    fun `can read an actual depth from table from real web page`() {
        WebPage.fromUrl("https://alert.valleywater.org/map?p=sensor&sid=4002&disc=f")
            .shouldNotBeNull()
            .findDepthTable()
            .shouldNotBeNull()
            .mostRecentDamReading()
            .shouldNotBeNull()
            .depth
            .shouldBePositive()
    }

uh-oh, it fails! Why does it fail? Because the webpage is populated by jquery queries. So ksoup won’t do it, we either need to identify the query and run it using ktor or somesuch, or we need to use browser automation like selenium which apparently does not exist in a kotlin multiplatform project yet (according to a brief search and questions to github copilot and jetbrains ai assistant anyway) so would require us to use expect/ actual and implement it separately in iOS, desktop, and android.

what seems the easiest is to use selenium, what seems the cleanest (so probably ends up easiest in the long run) is to identify the jquery call and replicate it to get the data.

Using the network tab of the chrome/ brave inpection, I find a request that gives a json response with the damn[sic] information! So what do I say about figuring this out without writing any tests? It was a spike. Sometimes you need to run experiments to figure out your direction. Now that I have this, I realize I will probably throw out all the web-scraping code and go to just query-request to that webpage. That I will do with tests again.

Question is, do we back out and start over or do we modify what we have? When do we modify, and when do we back up and try again? Seeing as we have not reached a working system, I believe we backup and try again. Anything we have done that we end up wanting to reuse will appear again with little or no trouble, but keeping old structures around for the supposed efficiency of reuse can falsely create structure that shouldn’t exist. So we will back out until we have no web-page based depth implementations.

Except we have config implementations interwoven there, maybe it would be easier to just delete the classes and push that. Do I commit git sin doing that? (sidenote, I wish there were a command git sin though I don’t know what it should do.

So I delete the commonMain/kotlin/web package and all dependent packages/ files

# Inbox
- [ ] decide whether depth is changed
  - [x] load last depth notified from config
  - [x] load change in depth to notify from config
  - [ ] read current depth from page and verify it is a double
    - [ ] get json query from page
      - [ ] send get request to url
      - [ ] convert response to json object
    - [ ] read depth from json query
      - [ ] get depth&dates from json object
      - [ ] sort by date
      - [ ] get last depth&date
      - [ ] extract depth from depth&date
      - [ ] verify depth is a positive double
      - [ ] validate date is recent enough
    - [ ] validate correct dam is being read
      - [ ] get location from json
      - [ ] compare location to known dam location
  - [ ] compute abs(last depth - current depth)
  - [ ] compare to change in depth to notify`
- [ ] notify user if depth is changed
- [ ] allow use****r to change parameters
- [ ] notify user if depth cannot be read
- [ ] check if a date is before or after another date
- [ ] check the time between two dates
- [ ] ui?

Now I am ready to start the json query part.

so lets start with a test:

    @Test
    fun `should get json from dam`() {
        val damReader = DamReader()

        val json = damReader.getReadings()

        json.shouldNotBeNull()
    }

after creating the stubs, we get a failure. This can be fixed by a stupid implementation:

class DamReader {
    fun getReadings(): Any? {
        return ""
    }
}

So we need a bit more specificity in our test:

class DamReaderGetsJsonTest {
    @Test
    fun `should get json from dam`() {
        val damReader = DamReader()

        val json = damReader.readingsResponseString()

        json.shouldNotBeNull()
            .shouldBeInstanceOf<String>()
            .trim()
            .shouldStartWith("{")
            .shouldEndWith("}")
    }
}

This should require some kind of json-like response at least, but the implementation is still stupid:

class DamReader {
    fun readingsResponseString(): String? {
        return "{}"
    }
}

So perhaps we are approaching this wrong. Why do we allow a null? Because we might query the wrong address. Lets add the query url to the DamReader and if it is notOurWebsite.com we get a null. But that forces us to put in a url to DamReader:

class DamReaderGetsJsonTest {
    private val damUrl = "https://alertdata.valleywater.org/sensorData/4002/range/2024-04-22/2024-04-23?includeRating=true&skipZeroValues=true"
    private val notTheDamUrl = "http://non-dam.com"
    
    @Test
    fun `should get json from dam`() {

        val damReader = DamReader(damUrl)

        val json = damReader.readingsResponseString()

        json.shouldNotBeNull()
            .shouldBeInstanceOf<String>()
            .trim()
            .shouldStartWith("{")
            .shouldEndWith("}")
    }

    @Test
    fun `should get null from non-dam`() {
        val damReader = DamReader(notTheDamUrl)

        val json = damReader.readingsResponseString()

        json.shouldBeNull()
    }
}

This fails because we do nothing with this information. Implementations that don’t work are: Ksoup because it doesn’t implement the connect, only has the parse. There is not easy way to perform a get of a non-http response. khttp because it is jvm only, and ktor seems overly complicated for the simple thing we need to do. So we either are at a place of using expect/ actual again with khttp for the jvm, or using ktor, or continuing to search.

Lets take another crack at ktor:

class DamReader(val damUrl: String) {

    fun readingsResponseString(): String? {
        return runBlocking{getResponse()}
    }

    private suspend fun getResponse(): String? {
        val client = HttpClient()

        try {
            val response = client.request("${damUrl}range/2024-04-22/2024-04-23") {
                method = HttpMethod.Get
            }
            return response.bodyAsText()
        } catch (e: Exception) {
            return null
        }
    }
}

This seems to work, we are now getting a json response from the actual web page. So now we need to see if we can find actual readings.

First step, convert to a json object. Lets start with a text json object (no need to go grab it every time).

class JSonReaderTranslatesTest {
    @Test
    fun `should translate json to depth`() {
        val json = """
            {"comments":[...],"metadata":{...},"columns":{"id":"Sensor ID","timestamp":"Date and Time (Pacific)","value":"Elevation (ft)","rating":"Storage (ac-ft)"},"data":[{"id":4002,
"timestamp":"2024-04-22T00:00:00-08:00","value":489.83,"rating":3590.98},
...}
        """.trimIndent()

        val depth = JSonReader(json).depth()

        depth.depth() shouldBeExactly 489.83
    }
}

This is too easily solved with another dumb implementation that just returns the expected depth. So we need another test, I’ll take this opportunity to test an edge case: what happens if the depth is an integer?:

class JSonReaderTranslatesTest {

    @Test
    fun `should translate json to depth`() {
        val json = """
        {
        "comments": [
            "NOTICE: These data readings are preliminary."
        ],
        "metadata": {
            "id": 4002
        },
        "columns": {
            "id": "Sensor ID",
            "timestamp": "Date and Time (Pacific)",
            "value": "Elevation (ft)",
            "rating": "Storage (ac-ft)"
        },
        "data": [
            {
                "id": 4002,
                "timestamp": "2024-04-22T00:00:00-08:00",
                "value": 489.83,
                "rating": 3590.98
            },
            {
                "id": 4002,
                "timestamp": "2024-04-22T00:15:00-08:00",
                "value": 489.84,
                "rating": 3592.67
            }
        ]
        }
        """.trimIndent()
        val depthReading: DepthReading = JSonReader(json).readDam()

        depthReading.shouldNotBeNull()
            .depth()
            .shouldBeExactly(489.83)
    }

    @Test
    fun `should translate json with no decimal to depth`() {
        val json = """
        {
        "comments": [
            "NOTICE: These data readings are preliminary."
        ],
        "metadata": {
            "id": 4002
        },
        "columns": {
            "id": "Sensor ID",
            "timestamp": "Date and Time (Pacific)",
            "value": "Elevation (ft)",
            "rating": "Storage (ac-ft)"
        },
        "data": [
            {
                "id": 4002,
                "timestamp": "2024-04-22T00:00:00-08:00",
                "value": 127,
                "rating": 3590.98
            },
            {
                "id": 4002,
                "timestamp": "2024-04-22T00:15:00-08:00",
                "value": 489.84,
                "rating": 3592.67
            }
        ]
        }
        """.trimIndent()
          val depth: DepthReading = JSonReader(json).readDam()

        depth.shouldNotBeNull()
            .depth()
            .shouldBeExactly(127.0)
    }

of course this fails with our dumb implementation. Now when I go to fix the implementation, I’m going to use the kotlinx serialization for this. This requires the kotlinx serialization compiler plugin to be included in my project, thanks github copilot! And then it requires a nested set of data classes to read the above json:

@Serializable
data class DamReport(
    val comments: List<String>,
    val metadata: Map<String, JsonElement>,
    val columns: Map<String, String>,
    val data: List<DepthReading>
)

@Serializable
data class DepthReading(
    val id: Int,
    val timestamp: String,
    val value: Double,
    val rating: Double
) {
    fun depth(): Double {
        return value
    }
}

one might think this is too big a step for one test. I felt fine with it because it felt like I would have to do a lot of convoluted shenanigans in order to avoid it, and it was all done with autocompletion so it felt simple. What do you think?

Kotlin Multiplatform with Compose - Dam Monitor

Lets actually do something! - changing direction, Ktor and kotlinx.serialization save us

Discussion about this post