Texting with UTF-8 Characters in C#

As technology expands the reach of our products and services, it’s important for us to create content to suite users from all over the world. As we primarily use Latin characters, creating content for certain languages like Japanese or Arabic that have a different alphabet is not as easy as typing them on a standard keyboard. This article will take us through how to send non-latin characters (emoji included) in text messages with C# using Twilio Programmable SMS service.

Before we start (System Check)

Besides basic knowledge in C#, to get started, you should have the following installed on your system:

• Windows Operating System
• Visual Studio 2017 with .NET Core, you can get the community edition.

Why UTF-8

The problem started when people realized that ASCII needed to be extended and other character sets had to be created to accommodate the alphabets of other languages. In the late 1980's there existed over 10 different character sets developed to cover the alphabets of languages like Cyrillic, Arabic, Hebrew, Turkish, to mention a few. It become cumbersome to keep track of and manage the different character sets. That’s why Unicode was developed. It integrated all these character sets into a single character set that boasts of over 100,000 different characters from more than 20 different alphabets. UTF-8 short for Universal Character Set Transformation Format 8 bit is simply an 8-bit representation of Unicode characters, it is the standard way to represent Unicode characters on computers.

There happens to be a really great article explaining Unicode and other encoding methods.

C# Escape characters

Now that we know a little more about UTF-8 and why it is so important, we’re going to learn how to use it in C#. To use UTF-8 characters in C# we use an inbuilt feature for parsing strings called escape characters. Escape characters let us play around with the format of strings and add additional functionality to them. For example, to store a string value in a variable, we use quotation marks to show that the text is string.

To store the sentence “Scripts over everything!” as a string in a variable, we write the following line of code.

String message = “Scripts over everything!”;
Console.Write(message);

When we run it, it works just fine and the message is printed.

However, if our text has quotation marks in it, we will get an error if we write the code below

String message = “Drew said “Scripts over everything” ”;
Console.Write(message);

The build fails as this would throw a syntax error.

To fix this we use escape characters. A list of C# escape characters can be found here.

For this case, we use \” , which lets us add quotation marks as part of the string. The code should look like this.

String message = "Drew said \"Scripts over everything\"";
Console.Write(message);

Similarly, to use non-Latin characters supported in UTF-8 we need to use the \u escape character followed by the escape sequence of the specific character(s) we want to use. We’ll get deeper into this in the next section.

Code

To get started we'll have to create a new project. Open Visual Studio 2017 and click New Project. You'll see a pop up ask what type of project we want to create., we need to select Console App under .NET Core. Let's name the app textio. Click Ok and you should have the following code on your screen after.

using System;

namespace textio
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Hello World!");
        }
    }
}

Next up we need to install the Twilio .NET package using NuGet or .NET CLI, Twilio has an awesome walk through on how to do this. In this tutorial, we'll use NuGet. Open the console using the Tools > NuGet Package Manager > Package Manager Console . After it initializes, type in Install-Package Twilio to install the package.

We’ll separate the solutions code into four parts:

• Adding Namespaces
• Adding our Credentials
• Sending our First Message
• Adding UTF-8

Adding Namespaces

In this section we need to make the compiler aware that we’re using certain names in our programs namespace. It’s also useful because it stops us from prepending namespaces when using their methods. For example, by including the line using System;, we can simply say Console.Write(message) instead of System.Console.Write(message). We include using Twilio; , similarly it gives us access to some functionality which in this case is the Twilio API. Your code should look something like this now.

// Get the twilio-csharp library from twilio.com/docs/libraries/csharp

using System;
using Twilio;
using Twilio.Rest.Api.V2010.Account;
using Twilio.Types;

namespace textio
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Hello World!");
        }
    }
}

Adding our Credentials

Next up we need to add our Twilio Account SID and Token to be able to use the Programmable Text service, you can get these in your Dashboard. We’ll also add your Twilio phone number, you create one or get an existing here. No worries if you don't have an account, Twilio let's us set one easily. Head to the Twilio Homepage and click Sign Up, enter your details ( it doesn't take more than 5 mins) and you should have a Twilio account open. Once you're logged in, head over to this Twilio page to get your new number setup.

Now that we have our Account SID, Token and Phone Number we delete the Console.WriteLine("Hello World!"); and replace it with the code below and replace the demo text with your actual credentials and your recipient's phone number.

// Find your Account Sid and Auth Token at twilio.com/console and add them below
const string accountSid = "<Your Twilio Account SID>";
const string authToken = "<Your Twilio Auth Token>";
TwilioClient.Init(accountSid, authToken);

// Add the number you want to send a text to here
var rec = "<Your recipient's phone number";

//Add your Twilio phone number here
var phone = "<Your Twilio Phone Number>";

Sending our First Message

In the last line of Main method ( right after var phone = "<Your Twilio Phone Number";, paste the code below. The string text stores the actual message we want to send, feel free to change it accordingly.

// Add the message you want to send here
string text = "Hello! Is it memes you're looking for?";

var to = new PhoneNumber(rec);
// Putting everything together and sending the text
var message = MessageResource.Create(
to,
from: new PhoneNumber(phone), body: text);

Console.WriteLine("\nYou just sent a text with Twilio *Mic Drop* ");
Console.WriteLine(message.Sid);
Console.Write("\nSuccess! Press any key to exit...");
Console.ReadKey(true);

The code block above straps everything together, MessageResourse.Create gets your number, the recipient and the message and sends the actual message using Twilio. The last few lines print a message and the message SID so you know that your message was sent. This is where the magic happens, let's test the code to see if you can receive text first. Run the application by pressing f5, you should receive a text message like this.

Adding UTF-8

Going back to the escape characters we talked about above, if we add the escape character u/ we should be able to use UTF-8 escape sequences. We can now add some Japanese, Hello in Japanese is こんにちは which has the escape sequence \u3053\u3093\u306B\u3061\u306F! Pretty cool, I used this website to convert the Japanese text to its escape sequence. If we change up the line the code that sends a message to this

string text = "\u3053\u3093\u306B\u3061\u306F”;

We should get lovely greeting in Japanese as show below.

Lets go a step further and add some emoji sparkle , I used this website to get some emoji escape sequences. The code should look like this

string text = "\u3053\u3093\u306B\u3061\u306F \u2728”;

We get this

I’m a huge movie buff. Replace this message with this code and see what you get!

string text = "\uD83D\uDC00 \uD83C\uDF72 \uD83D\uDC68\u200D\uD83C\uDF73 \uD83D\uDC45";

Can you guess what movie this is?

Conclusion

That’s it! We’re sending texts in any language supported by UTF-8. Today, we went through UTF-8, what it is and why it's important. We found out about escape sequences and how they help us manipulate strings. We particularly learned to use these escape sequences to add UTF-8 characters in our texts and sent some dope texts in Japanese. Hopefully this makes it easier for your products to reach a larger audience. You can find the complete project code on my GitHub.

Feel free to reach out to me with questions on my Twitter, or Instagram.