Blog

Introduction

During the development of Pesto, one of our software dedicated to data erase on hard disks/SSDs, we had to decide which approach to use in managing the individual processes of erase or writing data to the devices. After a brief (almost non-existent) discussion, we chose a client-server model and then immediately set to work implementing the various functions.

This choice was strongly influenced by the fact that it was convenient to perform disk operations remotely on another machine set up only to perform such actions.

So we have divided program tasks into "server tasks" (read/write operations on disks) and "client tasks" (receiving status/results of operations and graphical interface).

Having done that, we got stuck in the fundamental problem, that is the client-server communication protocol. Initially we developed a simple UDP protocol by using the socket libraries which, however, did not guarantee the complete reception of the commands by the server. So we switched to a TCP protocol, always using the socket libraries, but as the project progressed, small but serious protocol errors caused significant problems while complexity increased and thread management needed for the multiple operations the program must perform silently took over.

All of this led us to discover and employ the Twistedlibraries, saving us the trouble of having to write a protocol from scratch and allowing us to greatly speed up the writing of the other parts of the program.

What is Twisted and what is it for?

Twisted is an event-driven network programming framework that provides several useful tools for simplifying the writing of a network protocol.

With Twisted you can build echo servers, web servers, mail clients, SSH clients, …

What we are interested in, in this case, is to build a client-server communication protocol able to allow the client to send specific commands following specific events caused by actions that the user performs on the graphical interface or on the terminal.

Server side implemetation

For illustration purposes, let's build a small TCP server capable of receiving text strings and processing them to perform specific actions.

from twisted.internet import protocol, reactor
from twisted.protocols.basic import LineOnlyReceiver

class Server(LineOnlyReceiver):
    def connectionMade(self):
        self.sendLine(b"Connection with the server established!")
        print("Client connected.")

    def connectionLost(self, reason):
        print(f"Connection lost.")

    def dataReceived(self, data):
        data = data.decode('utf-8').rstrip('\n\r')
        print(f"Received data: {data}")
        if data == 'hello':
            self.sendLine(b"Hello! How are you?")
        if data == 'goodbye':
            self.sendLine(b"Noooo don't go awayyyyy...")
            self.transport.loseConnection()
        if 'write: ' in data:
            data = data.lstrip("write: ")
            if self.factory.saveText(data):
                self.sendLine(b"Message saved!")


class ServerFactory(protocol.Factory):
    protocol = Server

    def startFactory(self):
        self.fp = open('file.txt', 'a')

    def stopFactory(self):
        self.fp.close()

    def saveText(self, data):
        self.fp.write(data + '\n')
        self.fp.flush()
        return True


def main():
    reactor.listenTCP(1234, ServerFactory())
    reactor.run()


if __name__ == "__main__":
    main()

There are 3 key elements in the server code:

• Protocol → Server
• Factory → ServerFactory
• Reactor → reactor

The reactor constitutes the twisted event loop and handles all internal program events and messages.

The factory class is an object that allows us to separate the "local" functions from the communication functions that are left exclusively to the protocol.

Reactor

The reactor is called in the main() function with the listenTCP and run methods.

ThelistenTCP(porta, factory()) command builds the actual server, using a specific port that the user enters as an argument and calling a factory, which will take care of creating the protocol and establishing the connection. In our case, we'll use the port 1234 and the factory ServerFactory.

With run(), the Twisted event loop is started and it will run until the server is terminated or a catastrophic error occurs.

Factory

A factory object can be seen as a protocol factory which also performs all the operation associated with a single connection.

In this case we have used 2 "prefabricated" methods, startFactory and stopFactory, and a method we wrote to save the text to a file, saveText.

The startFactory(self) method is called every time a new connection is requested and executes all the code inside it once. In our case, a file.txt file is opened in append mode.

The stopFactory(self) method is instead used when the specific connection is dropped and, as forstartFactory it executes the code inside it only once. Here, it just closes the file.txt file.

The saveText(self, data)method, written by us, is called by the protocol when a message contains the string 'write: ' inside it and writes to the file.txt file everything that follows the command.

Protocol

The protocol is the object that manages all the communication with the connected client.

Three main methods are defined:

• connectionMade
• connectionLost
• dataReceived

The connectionMade method is called when a connection is established by executing the code inside it. In our case, a message is sent to the client confirming to the user that a connection has been made.

The connectionLostmethod, as the name suggests, executes the code it contains when the connection is terminated or lost.

The dataReceived method represents the core of the server, in a certain sense. In fact, inside this method it is established how the server should behave according to the messages it receives.

When the connection is established, the server sends a confirmation message to the client so that the user can determine if the connection has indeed been established.

If the client sends the command 'hello', the server will respond with 'Hello! How are you?

If the client sends a 'write:’ command, where <testo> verrà sostituito con un messaggio in particolare, il server salverà il testo inviato in un file locale e risponderà con ‘Testo salvato!’.

If the client sends the command ‘goodbye’, the server will respond with ‘Noooo don't go awayyyyy...’ and the connection will be closed.

Client side implementation

Now that we have created our server, we need to communicate with it via a client. As in the case of the server, we have 3 key objects:

• Protocol → Client
• Factory → ClientFactory
• Reactor Thread → ReactorThread

from twisted.protocols.basic import LineOnlyReceiver
from twisted.internet import reactor, protocol
from threading import Thread

receiver = None


class Client(LineOnlyReceiver):
    def connectionMade(self):
        global receiver
        receiver = self

    def connectionLost(self, reason):
        print("Connection lost.")

    def lineReceived(self, line):
        line = line.decode('utf-8')
        self.factory.update(line)

    def sendMsg(self, msg: str):
        if self is None:
            print("Cannot send message to server. No connection.")
        else:
            msg = msg.encode('utf-8')
            self.sendLine(msg)

    def disconnect(self):
        self.transport.loseConnection()


class ClientFactory(protocol.ClientFactory):
    protocol = Client

    def clientConnectionFailed(self, connector, reason):
        print("Connection failed.")

    def update(self, data):
        print(data)


class ReactorThread(Thread):
    def __init__(self, host: str, port: int):
        Thread.__init__(self)
        self.host = host
        self.port = port
        self.protocol = Client
        self.factory = ClientFactory()
        self.reactor = reactor

    def run(self):
        self.reactor.connectTCP(self.host, self.port, self.factory)
        self.reactor.run(installSignalHandlers=False)

    def stop(self):
        self.reactor.callFromThread(Client.disconnect, receiver)

    def send(self, msg: str):
        self.reactor.callFromThread(Client.sendMsg, receiver, msg)

    def reconnect(self):
        self.reactor.connectTCP(self.host, self.port, self.factory)
        self.reactor.callFromThread(Client.disconnect, receiver)

def main():
    host = "127.0.0.1"
    port = 1234
    r_thread = ReactorThread(host, port)
    r_thread.start()
    while True:
        cmd = input()
        if cmd == 'reconnect':
            r_thread.reconnect()
        else:
            r_thread.send(cmd)

if __name__ == '__main__':
    main()

In this program we will use the threading library to separate client-server communication from the user interface execution. This is because if we start the reactor in the main thread, it "takes over" the process and the user will have to wait for the program to finish entering commands, which is not really ideal for our purposes.

So we have 2 threads per client: one thread (the main thread) to insert the commands to send to the server and another thread for the reactor.

Reactor Thread

The reactor thread is defined in ReactorThread that is a Thread class which is instantiated at the beginning of the program.

When a new ReactorThread is instantiated is necessary to specify the address of the server to which you want to connect and the relative port. At thread startup the run method is executed and the reactor makes an attempt to connect to the server with the connectTCP method in which must be specified the IP address, the port and the client factory.

The other methods used in the example are:

• stop → Calls the protocol's disconnect method to make a disconnection.
• send → Calls the protocol's sendMsg method to send a message coming from the main thread.
• reconnect → Make a new connection attempt to the server (if the 'reconnect' command is sent from the main thread).reconnect‘)

Client Factory

As in the case of the server, here we have a client "factory" that can perform various local tasks leaving the protocol with the sole task of communicating with the server.

In our example there are 2 methods:

• clientConnectionFailed → Prints “Connection failed” if the client cannot establish a connection.
• update → Print the message received from the server on the terminal.

Protocol

In the protocol we define the methods useful for communication with the server:

• connectionMade → it “stores” the active protocol in a global receiver variable usable in the ReactorThread so you can send commands written on the main thread.
• connectionLost → Prints “Connection lost" if the connection is broken by the server.
• lineReceived → It is called when a message arrives from the server and sends it to the ClientFactory's update method to print the received message on the terminal.
• sendMsg → If you are connected to the server it sends the message from the input of the main thread, otherwise it prints an error stating that there is no connection.
• disconnect → Breaks the connection with the server.

Final considerations

Although it would have been very interesting to build a "low level" communication protocol with the socketlibraries, on the other hand we would have had to spend a lot of time and energy on this aspect, which in our case is not decisive because Pesto is not a project that requires such high efficiency in terms of communication between client and server.

Therefore, Twisted has proven to be a powerful tool capable of greatly speeding up software development.