Buckle up people, this is going to be a long one. After 8 days of research and endless hours of bug hunting, I’m happy to, finally, present this blog post to you.

Today we’ll take a look at Spring Boot with Couchbase and Neo4j, write a small Spring Boot application and experiment around with Couchbase and Neo4j. I really don’t know if this works as I want it to work, and the odds of this not working at all are quite high …but I’m a “glass-half-full” guy, so I’m going to try it anyways.

One of the tasks for my diploma project was the creation of an architecture that was both scalable and fast. A bit of context: I’m writing a social network for Initiative Interchange (a daughter organization of Rotary International). So, my partner had a couple requirements I had to fulfill with my architecture:

  • Have fast access times, no matter your physical location (aka. server distributed system)
  • Full text search
  • Find the connection between people (Person A is friends with Person B, Person B is friends with Person C, Person C is friends with Person D; How are Person A and Person D connected?)

I’ll admit it: I was absolutely clueless. I had only worked with relational databases before (sure…I did some smaller projects with MongoDB and Couchbase, but that doesn’t really count), so requirement #3 seemed completely out of scope for me. But thanks to the help of one of my teachers and a bit of research, I came up with a pretty solid architecture.

The architecture consists of three main parts:

  • The server, Java with Spring Boot
  • A database that…well…stores data (Couchbase)
  • A database that stores the relations between data (Neo4j)

This is basically a make or break situation right here. If I find out that this doesn’t work, I’d have to completely change the architecture of my diploma project….which would really screw things up for me. Nice.

Oh, and before I forget: Credit to baeldung, Denis Rosa, Jannik Hell and the Spring Guides for these blog posts (you guys helped a lot):
baeldung: Intro to Spring Data Couchbase, A Guide to Neo4J with Java, Introduction to Spring Data Neo4j
Denis Rosa: Couchbase with Spring-Boot and Spring Data
Jannik Hell: Using multiple datasources with Spring Boot and Spring Data 💈 ⇄🌱 ⇄ 💈
Spring Guides: Accessing Data with Neo4j

Setup

We’re going to build a really basic Spring Boot application. Nothing too special. We just have two domain objects: Users and Items. A user can own multiple items, an item can be owned by multiple users. So we just fire up IntelliJ, create a new project (with the Spring Initializr) and we’re good to go. I’m going to go with Java 11 because, at this point, I already got so used to the var keyword that I can’t do without🙃. Dependency wise we don’t really need much.

From Core, I got Lombok (cause, as you know, I’m lazy and won’t bother writing boilerplate code) and Cache (maybe I’ll play around with it at some point). I also got Web and grabbed Neo4j and Couchbase from NoSQL. Actuator is also pretty useful…so let’s get that as well.

I’ll host the databases on docker in a Ubuntu Server 18.04.1 LTS 64bit VM.

mkdir ~/couchbase
sudo docker run -d --name n4j --publish=7474:7474 --publish=7687:7687 --volume=$HOME/neo4j/data:/data neo4j
sudo docker run -d --name cb -p 8091-8094:8091-8094 -p 11210:11210 -v ~/couchbase:/opt/couchbase/var couchbase

Fun with configurations

One of the problems with Spring is, that you can’t actually use multiple database drivers in the same package. That’s just a limitation Spring has.
But fortunately for us, someone, way smarter than me, figured out how you can do the next best thing: separate repositories with different database drivers.
As described in Jannik Hell’s post, we need to create two different packages. One for Couchbase and one for Neo4j. This means, that we also need two different sets of domain objects.

So our project should contain two packages that look something like this:

neo_cb_project

I’ll just call all the Couchbase domain objects CbName and all the Neo4j domain objects NeoName. I’ll do that with all the repositories as well.

Neo4jConfiguration.java

@Configuration
@EnableNeo4jRepositories("at.simulevski.couchbaseneo4jdemo.neo4j.persistence")
@EnableTransactionManagement
public class Neo4jConfiguration {

    @Value("${neo4j.host}")
    private String host;

    @Value("${neo4j.username}")
    private String user;

    @Value("${neo4j.password}")
    private String password;

    @Bean
    public SessionFactory sessionFactory() {
        return new SessionFactory(configuration(), "at.simulevski.couchbaseneo4jdemo.neo4j.domain");
    }

    @Bean
    public Neo4jTransactionManager transactionManager() {
        return new Neo4jTransactionManager(sessionFactory());
    }

    @Bean
    public org.neo4j.ogm.config.Configuration configuration() {
        org.neo4j.ogm.config.Configuration configuration = new org.neo4j.ogm.config.Configuration.Builder()
                .uri(host)
                .credentials(user, password)
                .build();
        return configuration;
    }
}

CouchbaseConfiguration.java

@Configuration
@EnableCouchbaseRepositories
public class CouchbaseConfiguration extends AbstractCouchbaseConfiguration {

    @Value("${couchbase.host}")
    private String host;

    @Value("${couchbase.bucket}")
    private String bucket;

    @Value("${couchbase.username}")
    private String username;

    @Value("${couchbase.password}")
    private String password;

    @Override
    protected List<String> getBootstrapHosts() {
        return Arrays.asList(host);
    }

    @Override
    protected String getBucketName() {
        return bucket;
    }

    @Override
    protected String getUsername(){
        return username;
    }

    @Override
    protected String getBucketPassword() {
        return password;
    }
}

Of course, we still need to actually set the values for our configuration. We do that in the application.properties file.

application.properties

couchbase.host=192.168.0.121
couchbase.bucket=users
couchbase.username=demo
couchbase.password=demo123

neo4j.host=bolt://192.168.0.121
neo4j.username=demo
neo4j.password=demo123

Domain objects

Now that we have our database connections configured, let’s proceed to our domain objects. As mentioned before, we will need separate domain objects specific to each database.

So, our domain objects are pretty straightforward. Instead of @Table, we use @Document for Couchbase and @NodeEntity for Neo4j. Other than that, nothing really changes.

CbUser.java

@AllArgsConstructor
@NoArgsConstructor
@Builder

@Document
@Data
public class User {
    @Id
    @GeneratedValue(strategy = GenerationStrategy.UNIQUE)
    private String id;

    @NotNull
    @Field
    private String username;

    // We will fill this by hand
    private List<Item> items = new ArrayList<>();
}

CbItem.java

@AllArgsConstructor
@NoArgsConstructor
@Builder

@Document
@Data
public class Item {
    @Id
    @GeneratedValue(strategy = GenerationStrategy.UNIQUE)
    private String id;

    @NotNull
    @Field
    private String name;

    @Field
    private String description;
}

Neo4j on the other hand isn’t as easy. Apparently, Neo4j requires a connection object between two domain objects…but sometimes it also doesn’t (or I screwed up…which is probably more likely). I tried using a connection object (@RelationshipEntity) and it never worked for me. The example below works but I’m probably overlooking something here.

Anyways…figuring this out took me way longer than it should have…

NeoUser.java

@AllArgsConstructor
@NoArgsConstructor
@Builder

@Data
@NodeEntity
public class NeoUser {
    @Id
    @GeneratedValue
    Long id;

    @Index(unique = true)
    private String cbId;

    @Relationship(type = "OWNS")
    private List<NeoItem> items = new ArrayList<>();

    // Creates Neo4j relationship
    public void addItem(NeoItem item){
        items.add(item);
    }
    
    // Removes Neo4j relationship
    public void removeItem(NeoItem item) {
        items.remove(item);
    }
}

NeoItem.java

@AllArgsConstructor
@NoArgsConstructor
@Builder

@Data
@NodeEntity
public class NeoItem {
    @Id
    @GeneratedValue
    Long id;

    @Index(unique = true)
    private String cbId;
}

If life gives you databases, persist

The Neo4j repositories need to extend the Neo4jRepository interface, while the Couchbase repositories need to extend CrudRepository. The Couchbase repositories don’t really need much explaining, and work, pretty much, out-of-the-box.

CbUserRepository.java

@Repository
public interface CbUserRepository extends CrudRepository<User,String> {
    User findByUsername(String username);
}

CbItemRepository.java

@Repository
public interface CbItemRepository extends CrudRepository<Item,String> {
    Item findByName(String name);
}

Our Neo4j repositories are not as simple. For them to work properly, we need to write some Cypher code (the query language Neo4j uses), which, in turn, means that I have to learn it. Luckily, Cypher isn’t all that complicated, and a bit of messing around with it in the web-interface later, I got the gist of it.

NeoItemRepository.java

@Repository
public interface NeoItemRepository extends Neo4jRepository<NeoItem, Long> {
    @Query("MATCH (i:NeoItem{cbId:{cbId}}) DETACH DELETE i")
    Iterable<Long> deleteByCbId(@Param("cbId") String cbId);
    
    NeoItem findByCbId(String id);
}

NeoUserRepository.java

@Repository
public interface NeoUserRepository extends Neo4jRepository<NeoUser, Long> {
    @Query("MATCH (i:NeoUser{cbId:{cbId}}) DETACH DELETE i")
    Iterable<Long> deleteByCbId(@Param("cbId") String cbId);

    @Query("MATCH (u:NeoUser{cbId:{cbId}})-[r:OWNS]->(i:NeoItem) RETURN u,r,i")
    NeoUser findByCbId(@Param("cbId") String cbId);
}

Figuring out how to implement findByCbId was really freaking annoying. When I wrote the method definition without the @Query annotation, it sort-of worked. It returned the right NeoUser, but for some reason, the NeoUser had no NeoItems. Then I added the @Query annotation and it still didn’t work. Not even removing the connection entity worked. When I first tried them, my Cypher looked like this:

MATCH (u:NeoUser) WHERE u.cbId = {cbId} RETURN n

What I didn’t know, was that I had to return the user, the users connections and the child nodes for Spring to map the data correctly. Ugh🙄.

But the best part is, when I changed my Cypher code to this:

MATCH (u:NeoUser{cbId:{cbId}})-[r:OWNS]->(i:NeoItem) RETURN u,r,i

it also, initially, didn’t work! So I tried everything again until I finally figured out that I had to remove my connection entity.

When I removed the connection entity, it started working for some weird reason, but now I don’t have a navigation property to the user in my NeoItem class. Mind you, I tried all of these solutions twice (because with my first Cypher query, not even removing the connection entity worked).

It took me around 4 days to figure this stuff out…

Always at your service

Now that we have our repositories, we need to create the service layer. Remember, whenever we manipulate data, we need to access Couchbase and when we manipulate relationships, we use Neo4j. So, when we want to get a user with their items, we need to load all the relationships from Neo4j and load the items with the corresponding keys and add them to List<CbItem> items in the CbUser class.

The item service is fairly simple. We only do basic CRUD stuff there, so it’s not too exiting.

ItemService.java

@Service
public class ItemService {
    
    @Autowired
    private CbItemRepository cbItemRepository;

    @Autowired
    private NeoItemRepository neoItemRepository;

    @Transactional
    public boolean createItem(String name, String description) {
        if (cbItemRepository.findByName(name) != null){
            return false;
        }

        var item = CbItem.builder()
                .name(name)
                .description(description)
                .build();

        // Save the item in Couchbase...
        cbItemRepository.save(item);

        var neoItem = NeoItem.builder().cbId(item.getId()).build();
        // ...and in Neo4j
        neoItemRepository.save(neoItem);
        return true;
    }

    @Transactional
    public CbItem getItem(String name) {
        return cbItemRepository.findByName(name);
    }

    @Transactional
    public boolean deleteItem(String name) {
        var item = cbItemRepository.findByName(name);
        if (item == null){
            return false;
        }

        // Delete the item in Couchbase and in Neo4j
        cbItemRepository.deleteById(item.getId());
        neoItemRepository.deleteByCbId(item.getId());

        return true;
    }
}

The user service, on the other hand, is a bit more complex because this is where the magic happens. This is where we load the items from Couchbase, by keys and relations we store in Neo4j.

UserService.java

@Service
public class UserService {

    @Autowired
    private CbUserRepository cbUserRepository;

    @Autowired
    private CbItemRepository cbItemRepository;

    @Autowired
    private NeoUserRepository neoUserRepository;

    @Transactional
    public boolean createUser(String username) {
        if (cbUserRepository.findByUsername(username) != null){
            return false;
        }

        var user = CbUser.builder()
                .username(username)
                .build();
        
        // Save the user in Couchbase...
        cbUserRepository.save(user);

        var neoUser = NeoUser.builder().cbId(user.getId()).build();
        // ...and in Neo4j
        neoUserRepository.save(neoUser);
        return true;
    }

    @Transactional(readOnly = true)
    public CbUser getUser(String name) {
        var user = cbUserRepository.findByUsername(name);
        if (user == null){
            return null;
        }

        var neoUser = neoUserRepository.findByCbId(user.getId());
        var items = new ArrayList<CbItem>();
        // Load items from Couchbase
        for (var neoItem : neoUser.getItems()) {
            items.add(cbItemRepository.findById(neoItem.getCbId()).orElse(null));
        }

        user.setItems(items);

        return user;
    }

    @Transactional
    public boolean deleteUser(String name) {
        var user = cbUserRepository.findByUsername(name);
        if (user == null){
            return false;
        }

        // Delete the user in Couchbase and Neo4j
        cbUserRepository.deleteById(user.getId());
        neoUserRepository.deleteByCbId(user.getId());

        return true;
    }
}

And lastly, we need a service for our links.

LinkService.java

@Service
public class LinkService {

    @Autowired
    private CbUserRepository cbUserRepository;

    @Autowired
    private CbItemRepository cbItemRepository;

    @Autowired
    private NeoUserRepository neoUserRepository;

    @Autowired
    private NeoItemRepository neoItemRepository;

    @Transactional
    public boolean createItemLink(String userName, String itemName) {
        var user = cbUserRepository.findByUsername(userName);
        var item = cbItemRepository.findByName(itemName);
        if (user == null || item == null){
            return false;
        }

        var neoUser = neoUserRepository.findByCbId(user.getId());
        var neoItem = neoItemRepository.findByCbId(item.getId());

        neoUser.addItem(neoItem);
        neoUserRepository.save(neoUser);

        return true;
    }

    @Transactional
    public boolean deleteItemLink(String userName, String itemName) {
        var user = cbUserRepository.findByUsername(userName);
        var item = cbItemRepository.findByName(itemName);
        if (user == null || item == null){
            return false;
        }

        var neoUser = neoUserRepository.findByCbId(user.getId());
        var neoItem = neoItemRepository.findByCbId(item.getId());

        neoUser.removeItem(neoItem);
        neoUserRepository.save(neoUser);

        return true;
    }
}

Controller time (❍ᴥ❍ʋ)

The controllers really are quite simple. We want to create, read and delete users and items and, somehow, create a link between users and items. I came up with the following table:

URL Method Parameters Description
/user POST name Creates a user
/user/{name} GET - Gets a user
/user/{name} DELETE - Deletes a user
/item POST name, description Creates an item
/item/{name} GET - Gets an item
/item/{name} DELETE - Deletes an item
/link/{username}/{itemname} POST - Creates a link between user and item
/link/{username}/{itemname} DELETE - Deletes a link between user and item

So we end up with three controllers.

UserController.java

@Controller
public class UserController {

    @Autowired
    private UserService userService;

    @PostMapping("/user")
    public HttpEntity createUser(@RequestParam("name") String username){
        return userService.createUser(username) ? new ResponseEntity(HttpStatus.CREATED) : new ResponseEntity(HttpStatus.BAD_REQUEST);
    }

    @GetMapping("/user/{name}")
    public @ResponseBody
    CbUser getUser(@PathVariable("name") String name){
        return userService.getUser(name);
    }

    @DeleteMapping("/user/{name}")
    public HttpEntity deleteUser(@PathVariable("name") String name){
        return userService.deleteUser(name) ? new ResponseEntity(HttpStatus.OK) : new ResponseEntity(HttpStatus.BAD_REQUEST);
    }
}

ItemController.java

@Controller
public class ItemController {

    @Autowired
    private ItemService itemService;

    @PostMapping("/item")
    public HttpEntity createItem(@RequestParam("name") String name, @RequestParam("description") String description){
        return itemService.createItem(name, description) ? new ResponseEntity(HttpStatus.CREATED) : new ResponseEntity(HttpStatus.BAD_REQUEST);
    }

    @GetMapping("/item/{name}")
    public @ResponseBody
    CbItem getItem(@PathVariable("name") String name){
        return itemService.getItem(name);
    }

    @DeleteMapping("/item/{name}")
    public HttpEntity deleteItem(@PathVariable("name") String name){
        return itemService.deleteItem(name) ? new ResponseEntity(HttpStatus.OK) : new ResponseEntity(HttpStatus.BAD_REQUEST);
    }
}

LinkController.java

@Controller
public class LinkController {

    @Autowired
    private LinkService linkService;

    @PostMapping("/link/{user}/{item}")
    public HttpEntity createLink(@PathVariable("user") String user, @PathVariable("item") String item){
        return linkService.createItemLink(user,item) ? new ResponseEntity(HttpStatus.CREATED) : new ResponseEntity(HttpStatus.BAD_REQUEST);
    }

    @DeleteMapping("/link/{user}/{item}")
    public HttpEntity deleteLink(@PathVariable("user") String user, @PathVariable("item") String item){
        return linkService.deleteItemLink(user,item) ? new ResponseEntity(HttpStatus.OK) : new ResponseEntity(HttpStatus.BAD_REQUEST);
    }
}

Great! Let’s test this out with Postman.
I made the following requests:

POST localhost:8080/user?name=Ari
POST localhost:8080/item?name=Snickers&description=Und der Hunger ist gegessen
POST localhost:8080/item?name=Mars&description=Mars macht mobil
POST localhost:8080/link/Ari/Snickers
POST localhost:8080/link/Ari/Mars

Now let’s take a look at our databases.

Neo4j

neo_cb_neo_final

Couchbase

neo_cb_cb_final

Nice! That worked better than expected! Deleting also yields the expected behavior. Since we added custom delete queries in our Neo4j repositories, deleting a node also deletes its relationships.

Conclusion

I really like this architecture so far! Sure, initial setup was a bit complicated and it took me nearly a week to figure out how to properly use Neo4j, but now, that I had a closer look and “hAnDsOn ExPeRiEnCe” I feel really comfortable with these two persistence technologies. I really don’t know how well they’re going to scale (I know about XDCR on Couchbase but I need to investigate further for Neo4j) or how they compare in terms of speed (according to the teacher who helped me, this is the best solution for the requirements set by my partner). Eventually, I’ll maybe do some benchmarks myself, at some point.

Of course, you can find the entire project on GitHub.